Open In Colab

Facial Emotion Detection¶

ROBERT A, SLOAN

Problem Definition¶

The context: Why is this problem important to solve?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What are we trying to solve using data science?

About the dataset¶

The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:

‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.

PROBLEM STATEMENT:

CONTEXT: Computer Vision has use cases across almost all domains, from self-driving and evaluating the driver's experience in more automated and self-driving vehicles to the customer service industry across many types of jobs. The objective here is to build a range of different CNNs and test them to see what architecture works best for the problem of taking in an image and being able to recognize that A) there is a person in the image, but then also determine the emotional state of the person in the image.

Key Questions:

CNNs: Are they the right type of model for this task?

If CNNs are the right type of model, how are they constructed, and what kind of depth to the convolution layers is necessary and useful?

What is the best level of accuracy and overall performance that we can achieve before the model starts to become overfitted?

Transfer learning: How much can be gained by using pretrained models that are open-sourced? These models have been trained at no small expense on a large amount of labeled data that's time-intensive to collect and label. To reproduce them would be outside of what we are currently able to afford for the task, so can we gain from the hard work done by others, and if so, to what level?

We are trying to take in pixels from an image, value the pixel intensity, and then construct a neural network that can learn to see the world in terms of pixel values but be able to recognize the emergent features beyond that of just the pixel value. To see the human face and identify that one face has ears, and where there are ears, there is probably a face when looking at other images. Building on that to recognize the mouth and the facial features that change during different emotional states.

To accomplish this task we need the proper resources. NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2¶

In [70]:
gpu_info = !nvidia-smi
gpu_info = '\n'.join(gpu_info)
if gpu_info.find('failed') >= 0:
  print('Not connected to a GPU')
else:
  print(gpu_info)
Thu Feb 22 18:20:41 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.05             Driver Version: 535.104.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:00:04.0 Off |                    0 |
| N/A   53C    P0              28W /  70W |  14077MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
In [ ]:
!pip install tensorflow
!pip install keras-tuner
!pip install pandas
!pip install numpy
!pip install matplotlib
!pip install scikit-learn
!pip install imbalanced-learn
Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.15.0)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.5.26)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (16.0.6)
Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.25.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (67.7.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.9.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.36.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.60.1)
Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.2)
Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.42.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.27.0)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (1.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.5.2)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.31.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.0.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (1.3.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2024.2.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow) (2.1.5)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (3.2.2)
Collecting keras-tuner
  Downloading keras_tuner-1.4.6-py3-none-any.whl (128 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.9/128.9 kB 4.3 MB/s eta 0:00:00
Requirement already satisfied: keras in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.15.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (23.2)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from keras-tuner) (2.31.0)
Collecting kt-legacy (from keras-tuner)
  Downloading kt_legacy-1.0.5-py3-none-any.whl (9.6 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->keras-tuner) (2024.2.2)
Installing collected packages: kt-legacy, keras-tuner
Successfully installed keras-tuner-1.4.6 kt-legacy-1.0.5
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4)
Requirement already satisfied: numpy>=1.21.0 in /usr/local/lib/python3.10/dist-packages (from pandas) (1.25.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas) (1.16.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (1.25.2)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.49.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5)
Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.25.2)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (23.2)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.1)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2)
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.25.2)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.11.4)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.3.0)
Requirement already satisfied: imbalanced-learn in /usr/local/lib/python3.10/dist-packages (0.10.1)
Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.25.2)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.11.4)
Requirement already satisfied: scikit-learn>=1.0.2 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.2.2)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (1.3.2)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from imbalanced-learn) (3.3.0)

Mounting the Drive¶

NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.

In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive

Importing the Libraries¶

In [ ]:
import tensorflow as tf
In [ ]:
import zipfile
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import PIL
# Library for creating data paths
import os
from IPython.display import HTML
# Library for randomly selecting data points
import random
# Library for creating and showing plots
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
# Importing Deep Learning Libraries
from tensorflow.keras.utils import load_img
from tensorflow.keras.preprocessing.image import load_img, img_to_array
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import Dense, Input, Dropout, GlobalAveragePooling2D, Flatten, Conv2D, BatchNormalization, Activation, MaxPooling2D, LeakyReLU
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.optimizers import Adam, SGD, RMSprop, Adamax, AdamW
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
from tensorflow.keras.utils import to_categorical

# Importing all the required sub-modules from Keras
from keras.applications.vgg16 import VGG16
from tensorflow.keras.callbacks import LearningRateScheduler
from sklearn.utils import resample
from imblearn.over_sampling import SMOTE

Import the Facial_emotion_images data set from google drive as a zip file and extract contents into the colab workspace¶

In [ ]:
path = '/content/drive/My Drive/Deep learning/CapstoneProject/Facial_emotion_images.zip'

with zipfile.ZipFile(path, 'r') as zip_ref:
    zip_ref.extractall()

Let us load and unzip the data¶

In [ ]:
picture_size = 48
folder_path = "Facial_emotion_images/"

Note:

  • You must download the dataset from the link provided on Olympus and upload the same on your Google drive before executing the code in the next cell.
  • In case of any error, please make sure that the path of the file is correct as the path may be different for you.

Visualizing our Classes¶

Let's look at our classes.

Write down your observation for each class. What do you think can be a unique feature of each emotion, that separates it from the remaining classes?

Happy¶

In [ ]:
folder_path = 'Facial_emotion_images/'
num_happy = len(os.listdir(folder_path + "train/" + 'happy'))

# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
              f"<h3 style='color: #333;'>Training Data Information:</h3>" \
              f"<p style='color: blue;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))


expression1 = 'happy'
plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
    plt.subplot(2, 10, i)
    img = load_img(folder_path + "train/" + expression1 + "/" +
                  os.listdir(folder_path + "train/" + expression1)[i], target_size = (picture_size, picture_size))
    # Seperate the two expressions with a horizontal line
    if i == 8:
        plt.axhline()
    plt.imshow(img)

Training Data Information:

Number of training images for class 'happy' is: 3976

Observations and Insights:__

Surprised¶

In [ ]:
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))

# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
              f"<h3 style='color: #333;'>Training Data Information:</h3>" \
              f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))
expression = 'surprise'

plt.figure(figsize= (30
                     ,4))
for i in range(1, 21, 1):
    plt.subplot(2, 10, i)

    img = load_img(folder_path + "train/" + expression + "/" +
                  os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
    plt.imshow(img)

plt.show()

Training Data Information:

Number of training images for class 'surprise' is: 3173

Observations and Insights:__

Sad¶

In [ ]:
from IPython.display import HTML
import os

folder_path = 'Facial_emotion_images/'
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))

# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
              f"<h3 style='color: #333;'>Training Data Information:</h3>" \
              f"<p style='color: blue;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
              f"</div>"

display(HTML(output_html))
expression = 'sad'

plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
    plt.subplot(2, 10, i)

    img = load_img(folder_path + "train/" + expression + "/" +
                  os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
    plt.imshow(img)

plt.show()

Training Data Information:

Number of training images for class 'sad' is: 3982

Observations and Insights:__

Neutral¶

In [ ]:
folder_path = 'Facial_emotion_images/'
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))

# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
              f"<h3 style='color: #333;'>Training Data Information:</h3>" \
              f"<p style='color: blue;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))
expression = 'neutral'

plt.figure(figsize= (30,4))
for i in range(1, 21, 1):
    plt.subplot(2, 10, i)

    img = load_img(folder_path + "train/" + expression + "/" +
                  os.listdir(folder_path + "train/" + expression)[i], target_size = (picture_size, picture_size))
    plt.imshow(img)

plt.show()

Training Data Information:

Number of training images for class 'neutral' is: 3978

Observations and Insights:__

Checking Distribution of Classes¶

In [ ]:
import os
from keras.preprocessing.image import load_img
import matplotlib.pyplot as plt

# Assuming 'base_dir' is correctly set to your dataset directory
base_dir = 'Facial_emotion_images/train/'
categories = ['happy', 'sad', 'neutral', 'surprise']

fig, ax = plt.subplots(nrows=len(categories), ncols=4, figsize=(30, 8))  # Adjusted figsize for better visibility

for i, category in enumerate(categories):
    category_dir = os.path.join(base_dir, category)
    sample_images = os.listdir(category_dir)[:4]  # Get first 4 images of each category
    for j, image in enumerate(sample_images):
        img_path = os.path.join(category_dir, image)
        img = load_img(img_path, target_size=(48, 48), color_mode='grayscale')
        ax[i, j].imshow(img, cmap='gray')
        ax[i, j].axis('off')
        if j == 0:  # Mark the category name more prominently
            ax[i, j].set_ylabel(category.capitalize(), fontsize=14, labelpad=10, color='blue')

# Enhance layout and visibility
plt.tight_layout(pad=3.0)
plt.show()
In [ ]:
folder_path = 'Facial_emotion_images/'
num_surprise = len(os.listdir(folder_path + "train/" + 'surprise'))
num_happy= len(os.listdir(folder_path + "train/" + 'happy'))
num_neutral = len(os.listdir(folder_path + "train/" + 'neutral'))
num_sad = len(os.listdir(folder_path + "train/" + 'sad'))

# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; padding: 10px;text-align: center; font-size: 25px; border-radius: 10px;'>" \
              f"<h3 style='color: #333;'>Training Data Information:</h3>" \
              f"<p style='color: blue;'>Number of training images for class <b>'surprise'</b> is: <b>{num_surprise}</b></p>" \
              f"<p style='color: green;'>Number of training images for class <b>'neutral'</b> is: <b>{num_neutral}</b></p>" \
              f"<p style='color: orange;'>Number of training images for class <b>'happy'</b> is: <b>{num_happy}</b></p>" \
              f"<p style='color: red;'>Number of training images for class <b>'sad'</b> is: <b>{num_sad}</b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))
# Code to plot histogram
plt.figure(figsize = (30, 7))


# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]

%matplotlib inline

plt.subplot(2, 1, 1),

plt.bar(classes, num_images, color=['blue', 'orange', 'green', 'red'])
plt.xlabel('Class')
plt.ylabel('Number of Images')
plt.title('Number of Training Images per Class')
plt.show()

Training Data Information:

Number of training images for class 'surprise' is: 3173

Number of training images for class 'neutral' is: 3978

Number of training images for class 'happy' is: 3976

Number of training images for class 'sad' is: 3982

Think About It:

  • Are the classes equally distributed? If not, do you think the imbalance is too high? Will it be a problem as we progress?
  • Are there any Exploratory Data Analysis tasks that we can do here? Would they provide any meaningful insights?

OBSERVATION: The categorys sad neutral and happy all have the same ammount of images however the suprise category has less.

Observations and Insights:__

The data set is skewed and needs to have some data augmentation methods applied to equalize the data set.

In [ ]:
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]
plt.figure(figsize=(5, 5))

plt.pie(num_images, labels=classes, autopct='%1.1f%%', startangle=140, colors=['blue', 'orange', 'green', 'red'])
plt.title('Proportion of Each Class in the Dataset')
plt.show()
# Check the Data Set for imbalance
min_class_size = min(num_images)
max_class_size = max(num_images)
imbalance_ratio = max_class_size / min_class_size

import matplotlib.pyplot as plt

# Data
classes = ['surprise', 'happy', 'neutral', 'sad']
num_images = [3173, 3976, 3978, 3982]

%matplotlib inline


# Use HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
              f"<h3 style='color: red'>Imbalance Ratio: Categorical representation in the data set (Max Class Size / Min Class Size): 1.25 :</h3>" \
              f"<p style='color: #666;'><b>''</b>Imbalance Ratio between classes is:  <b>{imbalance_ratio:.2f}</b></p>" \
              f"<p style='color: red;'><b>''</b> Sad image representation is: <b> 26.4% </b></p>" \
              f"<p style='color: green;'><b>''</b>Neutral image representation is: <b> 26.3% </b></p>" \
              f"<p style='color: orange;'><b>''</b> Happy image representation is: <b> 26.3% </b></p>" \
              f"<p style='color: blue;'><b>''</b>Surprise image representation is: <b> 21.0% </b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))

Imbalance Ratio: Categorical representation in the data set (Max Class Size / Min Class Size): 1.25 :

''Imbalance Ratio between classes is: 1.25

'' Sad image representation is: 26.4%

''Neutral image representation is: 26.3%

'' Happy image representation is: 26.3%

''Surprise image representation is: 21.0%

In the initial EDA of the dataset, the categories of emotions appeared to be balanced except for 'surprise'. Digging into the statistical significance of the disparity, I have concluded the following

An imbalance ratio of 1.25 (Max Class Size / Min Class Size) suggests a relatively mild imbalance among the classes in the dataset.

Impact of Mild Imbalance Model Performance: Model Performance Modern neural networks, particularly those used for image classification tasks (like CNNs), are quite robust to mild class imbalances. An imbalance ratio close to 1 indicates that the classes are fairly well represented.

Learning Dynamics: The slight imbalance might not significantly skew the learning process, meaning the model can still learn to generalize well across all classes without heavy reliance on data balancing techniques.

To make sure that the mild inbalance is not going to impact the training process the following steps have been employed in the following models traing. Monitor Class-specific Metrics: Even with a mild imbalance, monitoring metrics like precision, recall, and F1-score for each class is good practice. This ensures that all classes are correctly learned by the model, and no class is systematically favored or ignored.

Mild Data Augmentation: While aggressive oversampling or synthetic data generation might not be necessary, employing data augmentation techniques rotation, flipping, scaling on the training set, can further help handle the slight imbalance's impact.

Creating our Data Loaders¶

In this section, we are creating data loaders that we will use as inputs to our Neural Network.

You have two options for the color_mode. You can set it to color_mode = 'rgb' or color_mode = 'grayscale'. You will need to try out both and see for yourself which one gives better performance.

In [90]:
from keras.preprocessing.image import ImageDataGenerator

# Define image size and paths
picture_size = 48
base_path = "Facial_emotion_images/"

train_path = base_path + "train/"
validation_path = base_path + "validation/"
test_path = base_path + "test/"

# Initialize the data generator for training data with data augmentation
train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)

# For validation and test sets, usually only rescaling is applied
test_val_datagen = ImageDataGenerator(rescale=1./255)
In [ ]:
train_generator = train_datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical'
)

validation_generator = test_val_datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical'
)

test_generator = test_val_datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical',
    shuffle=False  # Keep data in order for evaluation
)

# Getting the total Training images, Testing images and Validation images.
traingen = train_generator.n
testgen = test_generator.n
validationgen = validation_generator.n
# Using inline HTML to style the output
output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 25px; padding: 10px; border-radius: 10px;'>" \
              f"<p style='color: red;'><b>''</b>Total Training Images <b> {traingen}</b></p>" \
              f"<p style='color: black;'><b>''</b>Total Testing Images <b> {validationgen} </b></p>" \
              f"<p style='color: blue;'><b>'Data Set'</b>Total Validation Images <b>{testgen} </b></p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))

Functions for Classification_Report, Confusion_Matrix, Performance, Accuracy, Loss Plotting Functions¶

In [ ]:
#Defining the function for creating the custom classification report


def metrics_score(actual, predicted):

    from sklearn.metrics import classification_report

    from sklearn.metrics import confusion_matrix

    print(classification_report(actual, predicted))

    cm = confusion_matrix(actual, predicted)

    plt.figure(figsize = (8, 5))

    sns.heatmap(cm, annot = True,  fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)

    plt.ylabel('Actual')

    plt.xlabel('Predicted')

    plt.show()
In [82]:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report

def plot_model_performance_and_confusion_matrix(model, history, test_generator, metrics_function=None):
    # Plot training and validation accuracy
    plt.figure(figsize=(25, 5))
    plt.subplot(1, 3, 1)
    plt.plot(history.history['accuracy'], label='Training Accuracy')
    plt.plot(history.history['val_accuracy'], label='Validation Accuracy')
    plt.xlabel('Epoch')
    plt.ylabel('Accuracy')
    plt.title('Model Accuracy')
    plt.legend()

    # Plot training and validation loss
    plt.subplot(1, 3, 2)
    plt.plot(history.history['loss'], label='Training Loss')
    plt.plot(history.history['val_loss'], label='Validation Loss')
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.title('Model Loss')
    plt.legend()

    # Predict labels for the test set
    predicted_labels = np.argmax(model.predict(test_generator), axis=1)

    # Get the true labels for the test set
    true_labels = test_generator.classes

    # Optionally print class names
    class_names_list = list(test_generator.class_indices.keys())
    print("Class names in the dataset:", class_names_list)

    # Optionally calculate and display metrics
    if metrics_function is not None:
        metrics_function(true_labels, predicted_labels)

    # Plot the confusion matrix
    cm = confusion_matrix(true_labels, predicted_labels)
    plt.subplot(1, 3, 3)
    sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", xticklabels=class_names_list, yticklabels=class_names_list)
    plt.xlabel("Predicted Label")
    plt.ylabel("True Label")
    plt.title("Confusion Matrix")
    plt.show()
In [73]:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.metrics import confusion_matrix, classification_report

def plot_model_stats(model, history, test_generator, model_name):
    # Compute confusion matrix
    cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))

    # Plot the training and validation accuracy values
    plt.figure(figsize=(15, 5))
    plt.subplot(1, 3, 1)
    plt.plot(history.history['accuracy'])
    plt.plot(history.history['val_accuracy'])
    plt.title(f'{model_name} Accuracy')
    plt.ylabel('Accuracy')
    plt.xlabel('Epoch')
    plt.legend(['Train', 'Validation'], loc='upper left')

    # Plot the training and validation loss values
    plt.subplot(1, 3, 2)
    plt.plot(history.history['loss'])
    plt.plot(history.history['val_loss'])
    plt.title(f'{model_name} Loss')
    plt.ylabel('Loss')
    plt.xlabel('Epoch')
    plt.legend(['Train', 'Validation'], loc='upper left')

    # Plot confusion matrix
    plt.subplot(1, 3, 3)
    sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
    plt.xlabel("Predicted Label")
    plt.ylabel("True Label")
    plt.title(f"Confusion Matrix for {model_name}")
    plt.tight_layout()  # Adjust layout to not overlap figures
    plt.show()
In [75]:
def metrics_score(actual, predicted):

    from sklearn.metrics import classification_report

    from sklearn.metrics import confusion_matrix

    print(classification_report(actual, predicted))

    cm = confusion_matrix(actual, predicted)

    plt.figure(figsize = (8, 5))

    sns.heatmap(cm, annot = True,  fmt = '.0f', xticklabels = class_names_list, yticklabels = class_names_list)

    plt.ylabel('Actual')

    plt.xlabel('Predicted')

    plt.show()

Total Images in the categories and creating a weighted ouput for each emotion¶

In [ ]:
import os
from IPython.display import HTML, display

base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']

subset_counts = {subset: 0 for subset in subsets}

for subset in subsets:
    for emotion in classes:
        # Dir construction
        dir_path = os.path.join(base_path, subset, emotion)
        # Count files in directories and add to subset count
        if os.path.exists(dir_path):
            files_count = len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])
            subset_counts[subset] += files_count

output_html = f"<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 10px; border-radius: 10px;'>" \
              f"<p style='color: red;'><b>Total Training Images:</b> {subset_counts['train']}</p>" \
              f"<p style='color: black;'><b>Total Testing Images:</b> {subset_counts['test']} </p>" \
              f"<p style='color: blue;'><b>Total Validation Images:</b> {subset_counts['validation']} </p>" \
              f"</div>"

# Display the styled output
display(HTML(output_html))

Total Training Images: 15109

Total Testing Images: 128

Total Validation Images: 4977

In [ ]:
import os
from IPython.display import HTML, display

base_path = "Facial_emotion_images/"
subsets = ['train', 'test', 'validation']
classes = ['happy', 'sad', 'neutral', 'surprise']

class_counts = {emotion: 0 for emotion in classes}

for subset in subsets:
    for emotion in classes:
        dir_path = os.path.join(base_path, subset, emotion)
        if os.path.exists(dir_path):
            class_counts[emotion] += len([name for name in os.listdir(dir_path) if os.path.isfile(os.path.join(dir_path, name))])

# Generate HTML content
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Class Distribution</h3>"
for emotion, count in class_counts.items():
    output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {count}</p>"
output_html += "</div>"

# Display the styled output
display(HTML(output_html))

Class Distribution

Happy: 5833

Sad: 5153

Neutral: 5226

Surprise: 4002

In [ ]:
from sklearn.utils.class_weight import compute_class_weight
import numpy as np
from IPython.display import HTML, display


total_samples = sum(class_counts.values())
n_classes = len(classes)

# Compute class weights using sklearn's compute_class_weight function for better accuracy

y_integers = np.array([classes.index(emotion) for emotion in class_counts.keys() for _ in range(class_counts[emotion])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(y_integers), y=y_integers)
class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
# List the class weights in styled HTML
output_html = "<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>"
output_html += "<h3 style='color: #333;'>Computed Class Weights</h3>"
for emotion, weight in class_weights_dict.items():
    output_html += f"<p style='color: #333;'><b>{emotion.capitalize()}:</b> {weight:.2f}</p>"
output_html += "</div>"

# Display the styled output
display(HTML(output_html))

Computed Class Weights

Happy: 0.87

Sad: 0.98

Neutral: 0.97

Surprise: 1.26

Creating a weighted class call back for use during training to attempt to off set the class distribution dispairity¶

In [ ]:
from sklearn.utils.class_weight import compute_class_weight
import numpy as np

class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}
In [249]:
class_labels = np.array([classes.index(cls) for cls in classes for _ in range(class_counts[cls])])
class_weights = compute_class_weight(class_weight='balanced', classes=np.unique(class_labels), y=class_labels)
class_weights_dict = {i: weight for i, weight in enumerate(class_weights)}

class_weights_dict = {classes[i]: weight for i, weight in enumerate(class_weights)}
In [250]:
print("Class labels:", class_labels)
print("Class weights dictionary:", class_weights_dict)
print("Class weights:", class_weights)

# Adjust class weights to sum to 1
total_weight = sum(class_weights)
class_weights_adjusted = {cls: weight / total_weight for cls, weight in class_weights_dict.items()}
print("Class weights adjusted:", class_weights_adjusted)
Class labels: [0 0 0 ... 3 3 3]
Class weights dictionary: {'surprise': 1.262743628185907, 'happy': 0.8663637922166981, 'neutral': 0.96699196326062, 'sad': 0.9806908596933825}
Class weights: [1.26274363 0.86636379 0.96699196 0.98069086]
Class weights adjusted: {'surprise': 0.30973966105899836, 'happy': 0.21251125039569885, 'neutral': 0.23719443619558198, 'sad': 0.24055465234972084}
In [69]:
classes_sorted = sorted(classes)

class_weights_dict = {cls: weight for cls, weight in zip(classes_sorted, class_weights)}
In [ ]:
# Adjusting ClassWeights to ensure proper indicies assignment
class_weights_adjusted = {train_generator.class_indices[cls]: weight for cls, weight in class_weights_dict.items()}

Model Building¶

Creating the Base Neural Network¶

Think About It:

  • Are Convolutional Neural Networks the right approach? Should we have gone with Artificial Neural Networks instead?
  • What are the advantages of CNNs over ANNs and are they applicable here?

CNN_MODEL_1¶

In [ ]:
import tensorflow as tf
In [ ]:
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)
In [ ]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import Adam


cnn_model_1 = Sequential([
    # First Convolutional Block
    Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
    BatchNormalization(),
    Conv2D(64, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.25),

    # Second Convolutional Block
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.5),

    # Third Convolutional Block
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.5),

    # Fourth Convolutional Block
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    GlobalAveragePooling2D(),
    (Flatten()),
    # Dense Layers
    Dense(1024, activation='relu'),
    Dense(256, activation='relu'),
    Dropout(0.3),
    # Output
    Dense(128, activation='relu'),
    Dropout(0.1),
    Dense(64, activation='relu'),
    Dropout(0.25),
    Dense(4, activation='softmax')
])

cnn_model_summary = cnn_model_1.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 48, 48, 64)        1792      
                                                                 
 batch_normalization (Batch  (None, 48, 48, 64)        256       
 Normalization)                                                  
                                                                 
 conv2d_1 (Conv2D)           (None, 48, 48, 64)        36928     
                                                                 
 max_pooling2d (MaxPooling2  (None, 24, 24, 64)        0         
 D)                                                              
                                                                 
 dropout (Dropout)           (None, 24, 24, 64)        0         
                                                                 
 conv2d_2 (Conv2D)           (None, 24, 24, 128)       73856     
                                                                 
 batch_normalization_1 (Bat  (None, 24, 24, 128)       512       
 chNormalization)                                                
                                                                 
 conv2d_3 (Conv2D)           (None, 24, 24, 128)       147584    
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 12, 12, 128)       0         
 g2D)                                                            
                                                                 
 dropout_1 (Dropout)         (None, 12, 12, 128)       0         
                                                                 
 conv2d_4 (Conv2D)           (None, 12, 12, 256)       295168    
                                                                 
 batch_normalization_2 (Bat  (None, 12, 12, 256)       1024      
 chNormalization)                                                
                                                                 
 conv2d_5 (Conv2D)           (None, 12, 12, 256)       590080    
                                                                 
 batch_normalization_3 (Bat  (None, 12, 12, 256)       1024      
 chNormalization)                                                
                                                                 
 conv2d_6 (Conv2D)           (None, 12, 12, 256)       590080    
                                                                 
 max_pooling2d_2 (MaxPoolin  (None, 6, 6, 256)         0         
 g2D)                                                            
                                                                 
 dropout_2 (Dropout)         (None, 6, 6, 256)         0         
                                                                 
 conv2d_7 (Conv2D)           (None, 6, 6, 1024)        2360320   
                                                                 
 batch_normalization_4 (Bat  (None, 6, 6, 1024)        4096      
 chNormalization)                                                
                                                                 
 conv2d_8 (Conv2D)           (None, 6, 6, 1024)        9438208   
                                                                 
 global_average_pooling2d (  (None, 1024)              0         
 GlobalAveragePooling2D)                                         
                                                                 
 flatten (Flatten)           (None, 1024)              0         
                                                                 
 dense (Dense)               (None, 1024)              1049600   
                                                                 
 dense_1 (Dense)             (None, 256)               262400    
                                                                 
 dropout_3 (Dropout)         (None, 256)               0         
                                                                 
 dense_2 (Dense)             (None, 128)               32896     
                                                                 
 dropout_4 (Dropout)         (None, 128)               0         
                                                                 
 dense_3 (Dense)             (None, 64)                8256      
                                                                 
 dropout_5 (Dropout)         (None, 64)                0         
                                                                 
 dense_4 (Dense)             (None, 4)                 260       
                                                                 
=================================================================
Total params: 14894340 (56.82 MB)
Trainable params: 14890884 (56.80 MB)
Non-trainable params: 3456 (13.50 KB)
_________________________________________________________________

Compiling and Training the Model¶

In [71]:
cnn_model_1.compile(optimizer=AdamW(learning_rate=0.0005),  # Adjust learning rate as needed
              loss='categorical_crossentropy',
              metrics=['accuracy'])

from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau

early_stopping = EarlyStopping(monitor='accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.000002)
In [72]:
history = cnn_model_1.fit(
    train_generator,
    epochs=25,  # Adjust based on early stopping
    validation_data=validation_generator,
    callbacks=[early_stopping, reduce_lr],
    class_weight=class_weights_adjusted

)
Epoch 1/25
473/473 [==============================] - 33s 54ms/step - loss: 1.2893 - accuracy: 0.3542 - val_loss: 1.1666 - val_accuracy: 0.4744 - lr: 5.0000e-04
Epoch 2/25
473/473 [==============================] - 25s 52ms/step - loss: 1.0293 - accuracy: 0.5333 - val_loss: 0.9968 - val_accuracy: 0.5445 - lr: 5.0000e-04
Epoch 3/25
473/473 [==============================] - 25s 53ms/step - loss: 0.8731 - accuracy: 0.6343 - val_loss: 0.8473 - val_accuracy: 0.6383 - lr: 5.0000e-04
Epoch 4/25
473/473 [==============================] - 25s 53ms/step - loss: 0.7914 - accuracy: 0.6766 - val_loss: 0.9087 - val_accuracy: 0.6365 - lr: 5.0000e-04
Epoch 5/25
473/473 [==============================] - 25s 54ms/step - loss: 0.7443 - accuracy: 0.6993 - val_loss: 0.7194 - val_accuracy: 0.7079 - lr: 5.0000e-04
Epoch 6/25
473/473 [==============================] - 25s 54ms/step - loss: 0.7043 - accuracy: 0.7152 - val_loss: 0.7093 - val_accuracy: 0.7229 - lr: 5.0000e-04
Epoch 7/25
473/473 [==============================] - 25s 53ms/step - loss: 0.6734 - accuracy: 0.7295 - val_loss: 0.7230 - val_accuracy: 0.7163 - lr: 5.0000e-04
Epoch 8/25
473/473 [==============================] - 25s 53ms/step - loss: 0.6391 - accuracy: 0.7433 - val_loss: 0.7180 - val_accuracy: 0.7203 - lr: 5.0000e-04
Epoch 9/25
473/473 [==============================] - 25s 53ms/step - loss: 0.6195 - accuracy: 0.7541 - val_loss: 0.6271 - val_accuracy: 0.7511 - lr: 5.0000e-04
Epoch 10/25
473/473 [==============================] - 25s 53ms/step - loss: 0.5876 - accuracy: 0.7668 - val_loss: 0.6473 - val_accuracy: 0.7396 - lr: 5.0000e-04
Epoch 11/25
473/473 [==============================] - 25s 54ms/step - loss: 0.5608 - accuracy: 0.7767 - val_loss: 0.6750 - val_accuracy: 0.7326 - lr: 5.0000e-04
Epoch 12/25
472/473 [============================>.] - ETA: 0s - loss: 0.5481 - accuracy: 0.7880
Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
473/473 [==============================] - 25s 54ms/step - loss: 0.5475 - accuracy: 0.7883 - val_loss: 0.6758 - val_accuracy: 0.7324 - lr: 5.0000e-04
Epoch 13/25
473/473 [==============================] - 25s 53ms/step - loss: 0.4534 - accuracy: 0.8245 - val_loss: 0.6064 - val_accuracy: 0.7816 - lr: 1.0000e-04
Epoch 14/25
473/473 [==============================] - 25s 53ms/step - loss: 0.4201 - accuracy: 0.8327 - val_loss: 0.6151 - val_accuracy: 0.7824 - lr: 1.0000e-04
Epoch 15/25
473/473 [==============================] - 25s 54ms/step - loss: 0.3974 - accuracy: 0.8439 - val_loss: 0.6472 - val_accuracy: 0.7746 - lr: 1.0000e-04
Epoch 16/25
472/473 [============================>.] - ETA: 0s - loss: 0.3793 - accuracy: 0.8480
Epoch 16: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05.
473/473 [==============================] - 25s 54ms/step - loss: 0.3794 - accuracy: 0.8479 - val_loss: 0.6692 - val_accuracy: 0.7752 - lr: 1.0000e-04
Epoch 17/25
473/473 [==============================] - 25s 53ms/step - loss: 0.3476 - accuracy: 0.8654 - val_loss: 0.6500 - val_accuracy: 0.7808 - lr: 2.0000e-05
Epoch 18/25
473/473 [==============================] - 25s 53ms/step - loss: 0.3418 - accuracy: 0.8661 - val_loss: 0.6576 - val_accuracy: 0.7830 - lr: 2.0000e-05
Epoch 19/25
473/473 [==============================] - ETA: 0s - loss: 0.3340 - accuracy: 0.8690
Epoch 19: ReduceLROnPlateau reducing learning rate to 4.000000262749381e-06.
473/473 [==============================] - 25s 54ms/step - loss: 0.3340 - accuracy: 0.8690 - val_loss: 0.6585 - val_accuracy: 0.7840 - lr: 2.0000e-05
Epoch 20/25
473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8707 - val_loss: 0.6671 - val_accuracy: 0.7822 - lr: 4.0000e-06
Epoch 21/25
473/473 [==============================] - 25s 54ms/step - loss: 0.3277 - accuracy: 0.8656 - val_loss: 0.6694 - val_accuracy: 0.7824 - lr: 4.0000e-06
Epoch 22/25
472/473 [============================>.] - ETA: 0s - loss: 0.3257 - accuracy: 0.8731
Epoch 22: ReduceLROnPlateau reducing learning rate to 2e-06.
473/473 [==============================] - 25s 54ms/step - loss: 0.3255 - accuracy: 0.8732 - val_loss: 0.6722 - val_accuracy: 0.7820 - lr: 4.0000e-06
Epoch 23/25
473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8726 - val_loss: 0.6737 - val_accuracy: 0.7816 - lr: 2.0000e-06
Epoch 24/25
473/473 [==============================] - 25s 53ms/step - loss: 0.3273 - accuracy: 0.8723 - val_loss: 0.6743 - val_accuracy: 0.7820 - lr: 2.0000e-06
Epoch 25/25
473/473 [==============================] - 25s 53ms/step - loss: 0.3227 - accuracy: 0.8707 - val_loss: 0.6747 - val_accuracy: 0.7820 - lr: 2.0000e-06
In [ ]:
history = cnn_model_1.fit(
    train_generator,
    epochs=25,  # Adjust based on early stopping
    validation_data=validation_generator,
    callbacks=[early_stopping, reduce_lr],
    class_weight=class_weights_adjusted

)
Epoch 1/25
473/473 [==============================] - 27s 46ms/step - loss: 1.3672 - accuracy: 0.3026 - val_loss: 1.3214 - val_accuracy: 0.2648 - lr: 5.0000e-04
Epoch 2/25
473/473 [==============================] - 22s 46ms/step - loss: 1.2208 - accuracy: 0.3796 - val_loss: 1.2417 - val_accuracy: 0.3122 - lr: 5.0000e-04
Epoch 3/25
473/473 [==============================] - 22s 45ms/step - loss: 1.1498 - accuracy: 0.4201 - val_loss: 1.2578 - val_accuracy: 0.3255 - lr: 5.0000e-04
Epoch 4/25
473/473 [==============================] - 21s 45ms/step - loss: 1.1108 - accuracy: 0.4621 - val_loss: 1.1026 - val_accuracy: 0.5041 - lr: 5.0000e-04
Epoch 5/25
473/473 [==============================] - 21s 45ms/step - loss: 1.0652 - accuracy: 0.4964 - val_loss: 1.0543 - val_accuracy: 0.5188 - lr: 5.0000e-04
Epoch 6/25
473/473 [==============================] - 21s 45ms/step - loss: 0.9929 - accuracy: 0.5599 - val_loss: 0.9972 - val_accuracy: 0.5821 - lr: 5.0000e-04
Epoch 7/25
473/473 [==============================] - 21s 44ms/step - loss: 0.9165 - accuracy: 0.6112 - val_loss: 1.0254 - val_accuracy: 0.5600 - lr: 5.0000e-04
Epoch 8/25
473/473 [==============================] - 21s 44ms/step - loss: 0.8532 - accuracy: 0.6508 - val_loss: 0.7439 - val_accuracy: 0.6994 - lr: 5.0000e-04
Epoch 9/25
473/473 [==============================] - 21s 44ms/step - loss: 0.8271 - accuracy: 0.6621 - val_loss: 0.7456 - val_accuracy: 0.7036 - lr: 5.0000e-04
Epoch 10/25
473/473 [==============================] - 21s 44ms/step - loss: 0.8005 - accuracy: 0.6705 - val_loss: 0.6997 - val_accuracy: 0.7117 - lr: 5.0000e-04
Epoch 11/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7749 - accuracy: 0.6885 - val_loss: 0.6853 - val_accuracy: 0.7316 - lr: 5.0000e-04
Epoch 12/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7588 - accuracy: 0.6870 - val_loss: 0.6973 - val_accuracy: 0.7151 - lr: 5.0000e-04
Epoch 13/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7347 - accuracy: 0.7069 - val_loss: 0.6655 - val_accuracy: 0.7418 - lr: 5.0000e-04
Epoch 14/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7219 - accuracy: 0.7054 - val_loss: 0.6480 - val_accuracy: 0.7448 - lr: 5.0000e-04
Epoch 15/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7119 - accuracy: 0.7107 - val_loss: 0.6969 - val_accuracy: 0.7267 - lr: 5.0000e-04
Epoch 16/25
473/473 [==============================] - 21s 44ms/step - loss: 0.7008 - accuracy: 0.7190 - val_loss: 0.6514 - val_accuracy: 0.7338 - lr: 5.0000e-04
Epoch 17/25
472/473 [============================>.] - ETA: 0s - loss: 0.6900 - accuracy: 0.7220
Epoch 17: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
473/473 [==============================] - 21s 44ms/step - loss: 0.6894 - accuracy: 0.7222 - val_loss: 0.6523 - val_accuracy: 0.7509 - lr: 5.0000e-04
Epoch 18/25
473/473 [==============================] - 21s 44ms/step - loss: 0.6402 - accuracy: 0.7397 - val_loss: 0.5846 - val_accuracy: 0.7709 - lr: 1.0000e-04
Epoch 19/25
473/473 [==============================] - 21s 44ms/step - loss: 0.6118 - accuracy: 0.7498 - val_loss: 0.5854 - val_accuracy: 0.7687 - lr: 1.0000e-04
Epoch 20/25
473/473 [==============================] - 21s 44ms/step - loss: 0.6006 - accuracy: 0.7551 - val_loss: 0.5718 - val_accuracy: 0.7780 - lr: 1.0000e-04
Epoch 21/25
473/473 [==============================] - 21s 44ms/step - loss: 0.6130 - accuracy: 0.7484 - val_loss: 0.5883 - val_accuracy: 0.7675 - lr: 1.0000e-04
Epoch 22/25
473/473 [==============================] - 21s 44ms/step - loss: 0.5937 - accuracy: 0.7576 - val_loss: 0.5962 - val_accuracy: 0.7669 - lr: 1.0000e-04
Epoch 23/25
472/473 [============================>.] - ETA: 0s - loss: 0.5907 - accuracy: 0.7559
Epoch 23: ReduceLROnPlateau reducing learning rate to 2.0000000949949027e-05.
473/473 [==============================] - 21s 44ms/step - loss: 0.5912 - accuracy: 0.7557 - val_loss: 0.5798 - val_accuracy: 0.7673 - lr: 1.0000e-04
Epoch 24/25
473/473 [==============================] - 21s 44ms/step - loss: 0.5820 - accuracy: 0.7642 - val_loss: 0.5684 - val_accuracy: 0.7812 - lr: 2.0000e-05
Epoch 25/25
473/473 [==============================] - 21s 44ms/step - loss: 0.5749 - accuracy: 0.7667 - val_loss: 0.5712 - val_accuracy: 0.7788 - lr: 2.0000e-05

Saving the model in .h5 format¶

In [ ]:
cnn_model_1.save('cnn_model_1bestscores.h5')
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(

Converting the model into tensorflow.js format (json), for use in the web browser with tensorflow.js api.¶

Note: .h5 format is usable with python based backend such as flask however since the web is becoming dominated by javascript frame works such as react for front end client side user interfaces, and node.js backend and serverside functions converting the saved model to json format simplifys the deployment phase.

In [ ]:
!tensorflowjs_converter --input_format=keras /content/cnn_model_1bestscores.h5 /content/recallprecisionf1js
2024-02-20 00:53:00.430837: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-20 00:53:00.430894: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-20 00:53:00.432410: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-20 00:53:01.465249: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
In [84]:
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)

print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047
Test Loss: 0.6673559546470642
Test Accuracy: 0.8046875
In [83]:
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.87      0.81      0.84        32
           1       0.65      0.88      0.75        32
           2       0.81      0.69      0.75        32
           3       0.96      0.84      0.90        32

    accuracy                           0.80       128
   macro avg       0.82      0.80      0.81       128
weighted avg       0.82      0.80      0.81       128

In [86]:
from IPython.display import HTML, display

# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_1.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')

# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
        <p style='color: red;'><b>Test Loss:</b> CNN_MODEL_1 </p>
    <p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
    <p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""

# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 19ms/step - loss: 0.6674 - accuracy: 0.8047
Test Loss: 0.6673559546470642
Test Accuracy: 0.8046875

Test Loss: CNN_MODEL_1

Test Loss: 0.6674

Test Accuracy: 80.4688%

CNN_MODEL_2¶

  • Try out a slightly larger architecture
In [104]:
# Fixing the random seed generator to ensure random initilization numbers
np.random.seed(42)
import random
random.seed(42)
tf.random.set_seed(42)

Creating the second Convolutional Neural Network¶

In [105]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization

# Define the CNN model
cnn_model_updated = Sequential([
    # First Convolutional Block
    Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
    BatchNormalization(),
    Conv2D(64, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.2),  # Slightly reduced dropout

    # Second Convolutional Block
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.3),  # Adjusted dropout

    # Third Convolutional Block
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.4),  # Adjusted dropout

    # Fourth Convolutional Block
    Conv2D(512, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.4),  # Keeping consistent dropout

    # Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
    GlobalAveragePooling2D(),
    Dropout(0.4),  # Adjusted dropout

    # Simplified Dense Layers
    Dense(512, activation='relu'),
    Dropout(0.4),
    Dense(4, activation='softmax')  # Output layer for 4 classes
])

# Model Summary
cnn_model_updated.summary()

cnn_model_updated.compile(optimizer=Adamax(learning_rate=0.001),  # Adjusted learning rate
              loss='categorical_crossentropy',
              metrics=['accuracy'])

# Callbacks
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau

early_stopping = EarlyStopping(monitor='val_accuracy', patience=5, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=3, verbose=1, min_lr=0.0001)
Model: "sequential_13"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_87 (Conv2D)          (None, 48, 48, 64)        1792      
                                                                 
 batch_normalization_61 (Ba  (None, 48, 48, 64)        256       
 tchNormalization)                                               
                                                                 
 conv2d_88 (Conv2D)          (None, 48, 48, 64)        36928     
                                                                 
 max_pooling2d_48 (MaxPooli  (None, 24, 24, 64)        0         
 ng2D)                                                           
                                                                 
 dropout_71 (Dropout)        (None, 24, 24, 64)        0         
                                                                 
 conv2d_89 (Conv2D)          (None, 24, 24, 128)       73856     
                                                                 
 batch_normalization_62 (Ba  (None, 24, 24, 128)       512       
 tchNormalization)                                               
                                                                 
 conv2d_90 (Conv2D)          (None, 24, 24, 128)       147584    
                                                                 
 max_pooling2d_49 (MaxPooli  (None, 12, 12, 128)       0         
 ng2D)                                                           
                                                                 
 dropout_72 (Dropout)        (None, 12, 12, 128)       0         
                                                                 
 conv2d_91 (Conv2D)          (None, 12, 12, 256)       295168    
                                                                 
 batch_normalization_63 (Ba  (None, 12, 12, 256)       1024      
 tchNormalization)                                               
                                                                 
 conv2d_92 (Conv2D)          (None, 12, 12, 256)       590080    
                                                                 
 batch_normalization_64 (Ba  (None, 12, 12, 256)       1024      
 tchNormalization)                                               
                                                                 
 max_pooling2d_50 (MaxPooli  (None, 6, 6, 256)         0         
 ng2D)                                                           
                                                                 
 dropout_73 (Dropout)        (None, 6, 6, 256)         0         
                                                                 
 conv2d_93 (Conv2D)          (None, 6, 6, 512)         1180160   
                                                                 
 batch_normalization_65 (Ba  (None, 6, 6, 512)         2048      
 tchNormalization)                                               
                                                                 
 max_pooling2d_51 (MaxPooli  (None, 3, 3, 512)         0         
 ng2D)                                                           
                                                                 
 dropout_74 (Dropout)        (None, 3, 3, 512)         0         
                                                                 
 global_average_pooling2d_1  (None, 512)               0         
 2 (GlobalAveragePooling2D)                                      
                                                                 
 dropout_75 (Dropout)        (None, 512)               0         
                                                                 
 dense_37 (Dense)            (None, 512)               262656    
                                                                 
 dropout_76 (Dropout)        (None, 512)               0         
                                                                 
 dense_38 (Dense)            (None, 4)                 2052      
                                                                 
=================================================================
Total params: 2595140 (9.90 MB)
Trainable params: 2592708 (9.89 MB)
Non-trainable params: 2432 (9.50 KB)
_________________________________________________________________

Compiling and Training the Model¶

In [106]:
history = cnn_model_updated.fit(
    train_generator,
    epochs=50,  # Adjust based on early stopping
    validation_data=validation_generator,
    callbacks=[early_stopping, reduce_lr],
    class_weight=class_weights_adjusted

)
Epoch 1/50
473/473 [==============================] - 21s 39ms/step - loss: 1.5330 - accuracy: 0.3007 - val_loss: 1.5124 - val_accuracy: 0.2614 - lr: 0.0010
Epoch 2/50
473/473 [==============================] - 18s 38ms/step - loss: 1.3547 - accuracy: 0.3367 - val_loss: 1.3045 - val_accuracy: 0.4069 - lr: 0.0010
Epoch 3/50
473/473 [==============================] - 18s 37ms/step - loss: 1.2599 - accuracy: 0.3876 - val_loss: 1.2079 - val_accuracy: 0.4392 - lr: 0.0010
Epoch 4/50
473/473 [==============================] - 18s 39ms/step - loss: 1.1803 - accuracy: 0.4438 - val_loss: 1.0475 - val_accuracy: 0.5152 - lr: 0.0010
Epoch 5/50
473/473 [==============================] - 18s 38ms/step - loss: 1.1001 - accuracy: 0.5031 - val_loss: 1.1717 - val_accuracy: 0.5286 - lr: 0.0010
Epoch 6/50
473/473 [==============================] - 18s 38ms/step - loss: 0.9893 - accuracy: 0.5705 - val_loss: 0.8159 - val_accuracy: 0.6614 - lr: 0.0010
Epoch 7/50
473/473 [==============================] - 19s 41ms/step - loss: 0.9038 - accuracy: 0.6192 - val_loss: 0.8654 - val_accuracy: 0.6237 - lr: 0.0010
Epoch 8/50
473/473 [==============================] - 18s 38ms/step - loss: 0.8652 - accuracy: 0.6345 - val_loss: 0.8329 - val_accuracy: 0.6440 - lr: 0.0010
Epoch 9/50
473/473 [==============================] - 18s 38ms/step - loss: 0.8341 - accuracy: 0.6546 - val_loss: 0.6905 - val_accuracy: 0.7279 - lr: 0.0010
Epoch 10/50
473/473 [==============================] - 19s 40ms/step - loss: 0.8032 - accuracy: 0.6685 - val_loss: 0.7138 - val_accuracy: 0.7151 - lr: 0.0010
Epoch 11/50
473/473 [==============================] - 18s 39ms/step - loss: 0.7811 - accuracy: 0.6790 - val_loss: 0.7618 - val_accuracy: 0.6908 - lr: 0.0010
Epoch 12/50
473/473 [==============================] - ETA: 0s - loss: 0.7622 - accuracy: 0.6912
Epoch 12: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.
473/473 [==============================] - 18s 39ms/step - loss: 0.7622 - accuracy: 0.6912 - val_loss: 0.8096 - val_accuracy: 0.6610 - lr: 0.0010
Epoch 13/50
473/473 [==============================] - 18s 38ms/step - loss: 0.7197 - accuracy: 0.7091 - val_loss: 0.6167 - val_accuracy: 0.7490 - lr: 2.0000e-04
Epoch 14/50
473/473 [==============================] - 18s 39ms/step - loss: 0.7033 - accuracy: 0.7167 - val_loss: 0.6098 - val_accuracy: 0.7553 - lr: 2.0000e-04
Epoch 15/50
473/473 [==============================] - 18s 39ms/step - loss: 0.6967 - accuracy: 0.7189 - val_loss: 0.6111 - val_accuracy: 0.7494 - lr: 2.0000e-04
Epoch 16/50
473/473 [==============================] - 19s 39ms/step - loss: 0.6935 - accuracy: 0.7200 - val_loss: 0.6152 - val_accuracy: 0.7468 - lr: 2.0000e-04
Epoch 17/50
473/473 [==============================] - 19s 41ms/step - loss: 0.6796 - accuracy: 0.7239 - val_loss: 0.5981 - val_accuracy: 0.7599 - lr: 2.0000e-04
Epoch 18/50
473/473 [==============================] - 19s 41ms/step - loss: 0.6750 - accuracy: 0.7243 - val_loss: 0.5996 - val_accuracy: 0.7577 - lr: 2.0000e-04
Epoch 19/50
473/473 [==============================] - 19s 40ms/step - loss: 0.6636 - accuracy: 0.7290 - val_loss: 0.6017 - val_accuracy: 0.7615 - lr: 2.0000e-04
Epoch 20/50
473/473 [==============================] - ETA: 0s - loss: 0.6692 - accuracy: 0.7312
Epoch 20: ReduceLROnPlateau reducing learning rate to 0.0001.
473/473 [==============================] - 20s 41ms/step - loss: 0.6692 - accuracy: 0.7312 - val_loss: 0.6029 - val_accuracy: 0.7631 - lr: 2.0000e-04
Epoch 21/50
473/473 [==============================] - 19s 41ms/step - loss: 0.6599 - accuracy: 0.7342 - val_loss: 0.5979 - val_accuracy: 0.7593 - lr: 1.0000e-04
Epoch 22/50
473/473 [==============================] - 19s 39ms/step - loss: 0.6470 - accuracy: 0.7419 - val_loss: 0.6020 - val_accuracy: 0.7637 - lr: 1.0000e-04
Epoch 23/50
473/473 [==============================] - 18s 39ms/step - loss: 0.6533 - accuracy: 0.7359 - val_loss: 0.5943 - val_accuracy: 0.7641 - lr: 1.0000e-04
Epoch 24/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6441 - accuracy: 0.7425 - val_loss: 0.5866 - val_accuracy: 0.7681 - lr: 1.0000e-04
Epoch 25/50
473/473 [==============================] - 18s 39ms/step - loss: 0.6351 - accuracy: 0.7438 - val_loss: 0.5952 - val_accuracy: 0.7619 - lr: 1.0000e-04
Epoch 26/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6455 - accuracy: 0.7439 - val_loss: 0.5825 - val_accuracy: 0.7689 - lr: 1.0000e-04
Epoch 27/50
473/473 [==============================] - 19s 40ms/step - loss: 0.6396 - accuracy: 0.7444 - val_loss: 0.5929 - val_accuracy: 0.7609 - lr: 1.0000e-04
Epoch 28/50
473/473 [==============================] - 19s 40ms/step - loss: 0.6401 - accuracy: 0.7401 - val_loss: 0.5803 - val_accuracy: 0.7742 - lr: 1.0000e-04
Epoch 29/50
473/473 [==============================] - 19s 41ms/step - loss: 0.6328 - accuracy: 0.7465 - val_loss: 0.5784 - val_accuracy: 0.7744 - lr: 1.0000e-04
Epoch 30/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6305 - accuracy: 0.7479 - val_loss: 0.5822 - val_accuracy: 0.7715 - lr: 1.0000e-04
Epoch 31/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6291 - accuracy: 0.7468 - val_loss: 0.5808 - val_accuracy: 0.7754 - lr: 1.0000e-04
Epoch 32/50
473/473 [==============================] - 18s 39ms/step - loss: 0.6255 - accuracy: 0.7497 - val_loss: 0.5841 - val_accuracy: 0.7707 - lr: 1.0000e-04
Epoch 33/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6215 - accuracy: 0.7494 - val_loss: 0.5776 - val_accuracy: 0.7677 - lr: 1.0000e-04
Epoch 34/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6125 - accuracy: 0.7530 - val_loss: 0.5772 - val_accuracy: 0.7738 - lr: 1.0000e-04
Epoch 35/50
473/473 [==============================] - 18s 38ms/step - loss: 0.6201 - accuracy: 0.7522 - val_loss: 0.5783 - val_accuracy: 0.7736 - lr: 1.0000e-04
Epoch 36/50
472/473 [============================>.] - ETA: 0s - loss: 0.6184 - accuracy: 0.7519Restoring model weights from the end of the best epoch: 31.
473/473 [==============================] - 18s 38ms/step - loss: 0.6186 - accuracy: 0.7520 - val_loss: 0.5763 - val_accuracy: 0.7705 - lr: 1.0000e-04
Epoch 36: early stopping
In [111]:
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display

# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_updated.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')

# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
        <p style='color: red;'><b>Test Loss:</b> CNN_MODEL_updated </p>
    <p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
    <p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""

# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 10ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.82      0.84      0.83        32
           1       0.72      0.81      0.76        32
           2       0.79      0.69      0.73        32
           3       0.97      0.94      0.95        32

    accuracy                           0.82       128
   macro avg       0.82      0.82      0.82       128
weighted avg       0.82      0.82      0.82       128

4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203
Test Loss: 0.5187498927116394
Test Accuracy: 0.8203125

Test Loss: CNN_MODEL_updated

Test Loss: 0.5187

Test Accuracy: 82.0312%

Evaluating the Model on the Test Set¶

Evaluating the Model on the Test Set¶

Observations and Insights:__

While building the CNN models, i tried out various optimizers such as SGD, Adamax, AdamW, Adam, After running the model training several times with each and adjusting the learning rate along with the drop out

Model Architecture: The model is a deep convolutional neural network (CNN) designed with multiple convolutional blocks. Each block consists of convolutional layers with increasing filter sizes, BatchNormalization layers, MaxPooling for down-sampling, and Dropout layers to mitigate overfitting. The model concludes with dense layers and a softmax output layer for 4 classes. It uses GlobalAveragePooling2D to reduce the number of parameters and potentially decrease overfitting compared to using a Flatten layer.

Optimization and Callbacks: The model is compiled with the Adamax optimizer, a variant of Adam, with a low learning rate of 0.00018 to facilitate fine-grained updates to the model weights. Ituses callbacks such as EarlyStopping to halt training when validation accuracy stops to improve, ReduceLROnPlateau to decrease the learning rate when a metric has stopped improving, and ModelCheckpoint to save the model at its best performance based on validation accuracy.

Performance Metrics:

Accuracy: The overall accuracy on the validation set is 80%, indicating that the model correctly predicts the class for 80 out of every 100 images it is tested against. Precision, Recall, and F1-Score: The model shows varying levels of precision and recall across different classes: For the 'happy' class, it has a high precision of 0.87 but a slightly lower recall of 0.81, resulting in an F1-score of 0.84. This indicates it is quite good at identifying 'happy' faces but slightly conservative, missing some true 'happy' instances. The 'neutral' class has a precision of 0.65 with a high recall of 0.88, leading to an F1-score of 0.75. The model tends to label too many faces as 'neutral', but when it predicts 'neutral', it is often correct. For 'sad', precision is 0.81 and recall is 0.69, with an F1-score of 0.75. The model is relatively precise in identifying 'sad' faces but misses a significant portion of them. The 'surprise' class shows excellent precision at 0.96 and recall at 0.84, with an F1-score of 0.90. The model is highly effective at identifying 'surprise' faces with minimal false positives. Macro and Weighted Averages: Both macro and weighted averages across metrics hover around 0.82 for precision and 0.81 for the F1-score, suggesting a balanced performance across classes despite the variance in individual class metrics. Analysis: The model demonstrates good general performance with an overall accuracy of 80%. However, there's room for improvement, especially in balancing precision and recall across the different classes. The relatively lower precision for 'neutral' suggests a tendency to over-classify images as 'neutral', whereas the lower recall for 'sad' indicates missed detections of this emotion. The high performance on the 'surprise' class, with both high precision and recall, showcases the model's capability in identifying distinct features effectively.

Recommendations for Improvement:

Data Augmentation: To improve model generalization, especially for underperforming classes. Class Weights: Adjusting class weights during training could help address imbalances in precision and recall. Hyperparameter Tuning: Experimenting with different learning rates, optimizers, and dropout rates could enhance model performance. Model Architecture Adjustments: Tweaking the number of filters, layers, or incorporating additional regularization techniques might yield better results. Overall, cnn_model_2 shows promising results with a solid foundation for further optimization and tuning to enhance its ability to generalize across a range of facial expressions.

Observations and Insights:__ Some breif observaions, the models performed ok, reaching a highpoint of prediction accuracy of about 79 percent with cnn model 2. The precision and recall and f1 scores all are between 78 and 80 which is a good indicator. Still there is plenty of room for improvement and possibly with transferlearing the models will be able to get up to the 90 percent accuracy percent tile range.



Think About It:¶

  • Did the models have a satisfactory performance? If not, then what are the possible reasons?
  • Which Color mode showed better overall performance? What are the possible reasons? Do you think having 'rgb' color mode is needed because the images are already black and white?

Transfer Learning Architectures¶



In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.

Creating our Data Loaders for Transfer Learning Architectures¶

In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.

VGG16 Model¶

In [ ]:
# Set the random seed
np.random.seed(42)
In [ ]:
train_generator = train_datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical'
)

validation_generator = test_val_datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical'
)

test_generator = test_val_datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical',
    shuffle=False  # Keep data in order for evaluation
)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Importing the VGG16 Architecture¶

Model Building¶

  • Import VGG16 upto the layer of your choice and add Fully Connected layers on top of it.
In [ ]:
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras import Model

vgg = VGG16(include_top = False, weights = 'imagenet', input_shape = (48, 48, 3))
vgg.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
58889256/58889256 [==============================] - 2s 0us/step
Model: "vgg16"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]       0         
                                                                 
 block1_conv1 (Conv2D)       (None, 48, 48, 64)        1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 48, 48, 64)        36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 24, 24, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 24, 24, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 24, 24, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 12, 12, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 12, 12, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 6, 6, 256)         0         
                                                                 
 block4_conv1 (Conv2D)       (None, 6, 6, 512)         1180160   
                                                                 
 block4_conv2 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_conv3 (Conv2D)       (None, 6, 6, 512)         2359808   
                                                                 
 block4_pool (MaxPooling2D)  (None, 3, 3, 512)         0         
                                                                 
 block5_conv1 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv2 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_conv3 (Conv2D)       (None, 3, 3, 512)         2359808   
                                                                 
 block5_pool (MaxPooling2D)  (None, 1, 1, 512)         0         
                                                                 
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________

Compiling and Training the VGG16 Model_1¶

In [ ]:
np.random.seed(42)
In [ ]:
samples_per_class = {
    'surprise': 3173,
    'happy': 3976,
    'neutral': 3978,
    'sad': 3982,
}

# Calculate total samples
total_samples = sum(samples_per_class.values())

# Calculate class weights
class_weights = {class_id: total_samples/(len(samples_per_class)*num_samples)
                 for class_id, num_samples in enumerate(samples_per_class.values())}

print(class_weights)
{0: 1.1904349196344153, 1: 0.9500125754527163, 2: 0.949534942182001, 3: 0.9485811150175791}
In [ ]:
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"

# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)

# Prepare the data generators
train_generator = datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

validation_generator = datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,  # Keep the same batch size for simplicity
    class_mode='categorical')

test_generator = datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    shuffle=False)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Compiling and Training the VGG16 Model¶

In [ ]:
# Set the random seed
np.random.seed(42)
In [ ]:
from tensorflow.keras.applications.vgg16 import VGG16
from tensorflow.keras.layers import Dense, Flatten, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import AdamW, SGD, Adamax

# Set the random seed
np.random.seed(42)
#

# Load the VGG16 model
vgg = VGG16(include_top=False, weights='imagenet', input_shape=(48, 48, 3))

# Freeze all layers in the base VGG16 model
for layer in vgg.layers:
    layer.trainable = False

# Select the output of the 'block3_pool' layer
x = vgg.get_layer('block3_pool').output

# Continue adding your custom layers from here
x = Flatten()(x)  # Flatten the output of 'block3_pool'
x = Dense(512, activation='relu')(x)  # Custom dense layer
x = Dropout(0.1)(x)  # Dropout layer
x = Dense(1024, activation='relu')(x)  # Another dense layer
x = Dropout(0.2)(x)  # Dropout layer
x = Dense(32, activation='relu')(x)


# Final layer with softmax for classification
predictions = Dense(4, activation='softmax')(x)  # Assuming 4 classes for emotions

# Create the final model using VGG16 input and your custom output
model = Model(inputs=vgg.input, outputs=predictions)

# Compile the model
model.compile(optimizer=Adamax(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()
Model: "model_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_5 (InputLayer)        [(None, 48, 48, 3)]       0         
                                                                 
 block1_conv1 (Conv2D)       (None, 48, 48, 64)        1792      
                                                                 
 block1_conv2 (Conv2D)       (None, 48, 48, 64)        36928     
                                                                 
 block1_pool (MaxPooling2D)  (None, 24, 24, 64)        0         
                                                                 
 block2_conv1 (Conv2D)       (None, 24, 24, 128)       73856     
                                                                 
 block2_conv2 (Conv2D)       (None, 24, 24, 128)       147584    
                                                                 
 block2_pool (MaxPooling2D)  (None, 12, 12, 128)       0         
                                                                 
 block3_conv1 (Conv2D)       (None, 12, 12, 256)       295168    
                                                                 
 block3_conv2 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_conv3 (Conv2D)       (None, 12, 12, 256)       590080    
                                                                 
 block3_pool (MaxPooling2D)  (None, 6, 6, 256)         0         
                                                                 
 flatten_7 (Flatten)         (None, 9216)              0         
                                                                 
 dense_42 (Dense)            (None, 512)               4719104   
                                                                 
 dropout_55 (Dropout)        (None, 512)               0         
                                                                 
 dense_43 (Dense)            (None, 1024)              525312    
                                                                 
 dropout_56 (Dropout)        (None, 1024)              0         
                                                                 
 dense_44 (Dense)            (None, 32)                32800     
                                                                 
 dense_45 (Dense)            (None, 4)                 132       
                                                                 
=================================================================
Total params: 7012836 (26.75 MB)
Trainable params: 5277348 (20.13 MB)
Non-trainable params: 1735488 (6.62 MB)
_________________________________________________________________
In [ ]:
from tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping

reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)

# Train the model
history = model.fit(
    train_generator,
    epochs=75,
    validation_data=validation_generator,
    callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/75
473/473 [==============================] - 10s 17ms/step - loss: 1.8805 - accuracy: 0.2998 - val_loss: 1.3816 - val_accuracy: 0.2620 - lr: 2.5000e-05
Epoch 2/75
473/473 [==============================] - 8s 17ms/step - loss: 1.3644 - accuracy: 0.3341 - val_loss: 1.3035 - val_accuracy: 0.3709 - lr: 2.5000e-05
Epoch 3/75
473/473 [==============================] - 8s 16ms/step - loss: 1.2823 - accuracy: 0.4023 - val_loss: 1.2123 - val_accuracy: 0.4571 - lr: 2.5000e-05
Epoch 4/75
473/473 [==============================] - 8s 16ms/step - loss: 1.2114 - accuracy: 0.4331 - val_loss: 1.1272 - val_accuracy: 0.5120 - lr: 2.5000e-05
Epoch 5/75
473/473 [==============================] - 8s 16ms/step - loss: 1.1606 - accuracy: 0.4671 - val_loss: 1.1128 - val_accuracy: 0.5310 - lr: 2.5000e-05
Epoch 6/75
473/473 [==============================] - 8s 16ms/step - loss: 1.1103 - accuracy: 0.4885 - val_loss: 1.0563 - val_accuracy: 0.5429 - lr: 2.5000e-05
Epoch 7/75
473/473 [==============================] - 7s 16ms/step - loss: 1.0590 - accuracy: 0.5169 - val_loss: 0.9952 - val_accuracy: 0.5626 - lr: 2.5000e-05
Epoch 8/75
473/473 [==============================] - 8s 16ms/step - loss: 1.0040 - accuracy: 0.5544 - val_loss: 0.9309 - val_accuracy: 0.6168 - lr: 2.5000e-05
Epoch 9/75
473/473 [==============================] - 8s 16ms/step - loss: 0.9363 - accuracy: 0.5885 - val_loss: 0.8933 - val_accuracy: 0.6359 - lr: 2.5000e-05
Epoch 10/75
473/473 [==============================] - 8s 16ms/step - loss: 0.8996 - accuracy: 0.6125 - val_loss: 0.8726 - val_accuracy: 0.6418 - lr: 2.5000e-05
Epoch 11/75
473/473 [==============================] - 8s 16ms/step - loss: 0.8622 - accuracy: 0.6273 - val_loss: 0.8536 - val_accuracy: 0.6546 - lr: 2.5000e-05
Epoch 12/75
473/473 [==============================] - 8s 16ms/step - loss: 0.8274 - accuracy: 0.6489 - val_loss: 0.8336 - val_accuracy: 0.6612 - lr: 2.5000e-05
Epoch 13/75
473/473 [==============================] - 8s 17ms/step - loss: 0.8026 - accuracy: 0.6589 - val_loss: 0.8221 - val_accuracy: 0.6620 - lr: 2.5000e-05
Epoch 14/75
473/473 [==============================] - 8s 16ms/step - loss: 0.7751 - accuracy: 0.6736 - val_loss: 0.8141 - val_accuracy: 0.6775 - lr: 2.5000e-05
Epoch 15/75
473/473 [==============================] - 7s 16ms/step - loss: 0.7490 - accuracy: 0.6816 - val_loss: 0.8283 - val_accuracy: 0.6739 - lr: 2.5000e-05
Epoch 16/75
473/473 [==============================] - 8s 16ms/step - loss: 0.7240 - accuracy: 0.6998 - val_loss: 0.7986 - val_accuracy: 0.6801 - lr: 2.5000e-05
Epoch 17/75
473/473 [==============================] - 8s 16ms/step - loss: 0.6963 - accuracy: 0.7090 - val_loss: 0.7856 - val_accuracy: 0.6856 - lr: 2.5000e-05
Epoch 18/75
473/473 [==============================] - 7s 16ms/step - loss: 0.6791 - accuracy: 0.7133 - val_loss: 0.7866 - val_accuracy: 0.6845 - lr: 2.5000e-05
Epoch 19/75
473/473 [==============================] - 8s 17ms/step - loss: 0.6501 - accuracy: 0.7321 - val_loss: 0.7696 - val_accuracy: 0.6922 - lr: 2.5000e-05
Epoch 20/75
473/473 [==============================] - 7s 16ms/step - loss: 0.6265 - accuracy: 0.7390 - val_loss: 0.8019 - val_accuracy: 0.6894 - lr: 2.5000e-05
Epoch 21/75
473/473 [==============================] - 8s 16ms/step - loss: 0.6100 - accuracy: 0.7475 - val_loss: 0.7730 - val_accuracy: 0.6940 - lr: 2.5000e-05
Epoch 22/75
473/473 [==============================] - 8s 16ms/step - loss: 0.5893 - accuracy: 0.7551 - val_loss: 0.7784 - val_accuracy: 0.6920 - lr: 2.5000e-05
Epoch 23/75
473/473 [==============================] - 8s 16ms/step - loss: 0.5662 - accuracy: 0.7686 - val_loss: 0.7709 - val_accuracy: 0.6994 - lr: 2.5000e-05
Epoch 24/75
473/473 [==============================] - 7s 16ms/step - loss: 0.5509 - accuracy: 0.7749 - val_loss: 0.7765 - val_accuracy: 0.6946 - lr: 2.5000e-05
Epoch 25/75
473/473 [==============================] - 8s 16ms/step - loss: 0.5189 - accuracy: 0.7917 - val_loss: 0.7767 - val_accuracy: 0.6966 - lr: 5.0000e-06
Epoch 26/75
473/473 [==============================] - 8s 16ms/step - loss: 0.5130 - accuracy: 0.7920 - val_loss: 0.7657 - val_accuracy: 0.7024 - lr: 5.0000e-06
Epoch 27/75
473/473 [==============================] - 8s 16ms/step - loss: 0.5047 - accuracy: 0.7953 - val_loss: 0.7609 - val_accuracy: 0.7018 - lr: 5.0000e-06
Epoch 28/75
473/473 [==============================] - 8s 16ms/step - loss: 0.4956 - accuracy: 0.8018 - val_loss: 0.7666 - val_accuracy: 0.7036 - lr: 5.0000e-06
Epoch 29/75
473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.7997 - val_loss: 0.7728 - val_accuracy: 0.6998 - lr: 5.0000e-06
Epoch 30/75
473/473 [==============================] - 7s 16ms/step - loss: 0.4924 - accuracy: 0.8024 - val_loss: 0.7701 - val_accuracy: 0.7028 - lr: 5.0000e-06
Epoch 31/75
473/473 [==============================] - 8s 16ms/step - loss: 0.4835 - accuracy: 0.8064 - val_loss: 0.7681 - val_accuracy: 0.7018 - lr: 5.0000e-06
Epoch 32/75
473/473 [==============================] - 7s 16ms/step - loss: 0.4756 - accuracy: 0.8108 - val_loss: 0.7696 - val_accuracy: 0.7012 - lr: 5.0000e-06
Epoch 33/75
473/473 [==============================] - 8s 16ms/step - loss: 0.4751 - accuracy: 0.8079 - val_loss: 0.7710 - val_accuracy: 0.7026 - lr: 1.0000e-06
Epoch 34/75
473/473 [==============================] - 7s 15ms/step - loss: 0.4773 - accuracy: 0.8082 - val_loss: 0.7680 - val_accuracy: 0.7032 - lr: 1.0000e-06
In [ ]:
from sklearn.metrics import confusion_matrix, classification_report
cm = confusion_matrix(test_generator.classes, np.argmax(model.predict(test_generator), axis=1))

# Confusion matrix
print("Confusion Matrix:")
print(cm)

# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('VGG_Model_1 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('VGG_Model_1 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for VGG_MODEL_1")
plt.show()
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(model.predict(test_generator), axis=1)

# Get the true labels for the test set
true_labels = test_generator.classes

# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)

# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)

# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 11ms/step
Confusion Matrix:
[[25  6  0  1]
 [ 0 28  3  1]
 [ 3  8 18  3]
 [ 0  2  2 28]]
4/4 [==============================] - 0s 13ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
              precision    recall  f1-score   support

           0       0.89      0.78      0.83        32
           1       0.64      0.88      0.74        32
           2       0.78      0.56      0.65        32
           3       0.85      0.88      0.86        32

    accuracy                           0.77       128
   macro avg       0.79      0.77      0.77       128
weighted avg       0.79      0.77      0.77       128

The customized VGG16 model, adapted for emotion classification into four categories (happy, neutral, sad, surprise), demonstrates an overall accuracy of 77%. The model shows varying precision and recall across classes, with notable strengths in identifying 'surprise' emotions accurately. Despite its solid foundation, there's room for improvement, especially in balancing performance across all classes. Recommendations include data augmentation, fine-tuning of VGG16 layers, optimization of hyperparameters, and adjustment of class weights to enhance generalization and performance uniformly across categories.

In [ ]:
 

Think About It:

  • What do you infer from the general trend in the training performance?
  • Is the training accuracy consistently improving?
  • Is the validation accuracy also improving similarly?

**Observations and Insights:__It is seeming to prove to be difficult to get the model performance above 75-80 percent accuracy. Additionally, The Category of neutral is often what gets predicted when the actual class is that of 'sad', The largest confusion point seems to be a sad face being miss classified for 'neutral'



ResNet V2 Model¶

Model Building¶

  • Import Resnet v2 upto the layer of your choice and add Fully Connected layers on top of it.
In [ ]:
# Set the random Seed
np.random.seed(42)
In [ ]:
from tensorflow.keras.applications.resnet50 import ResNet50, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, Flatten, Dropout
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.resnet50 import preprocess_input, ResNet50
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
In [ ]:
import tensorflow as tf
import tensorflow.keras.applications as ap
from tensorflow.keras import Model

Resnet = ap.ResNet101(include_top = False, weights = "imagenet", input_shape=(48, 48, 3))
Resnet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet101_weights_tf_dim_ordering_tf_kernels_notop.h5
171446536/171446536 [==============================] - 6s 0us/step
Model: "resnet101"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]          0         []                            
                                                                                                  
 conv1_pad (ZeroPadding2D)   (None, 54, 54, 3)            0         ['input_1[0][0]']             
                                                                                                  
 conv1_conv (Conv2D)         (None, 24, 24, 64)           9472      ['conv1_pad[0][0]']           
                                                                                                  
 conv1_bn (BatchNormalizati  (None, 24, 24, 64)           256       ['conv1_conv[0][0]']          
 on)                                                                                              
                                                                                                  
 conv1_relu (Activation)     (None, 24, 24, 64)           0         ['conv1_bn[0][0]']            
                                                                                                  
 pool1_pad (ZeroPadding2D)   (None, 26, 26, 64)           0         ['conv1_relu[0][0]']          
                                                                                                  
 pool1_pool (MaxPooling2D)   (None, 12, 12, 64)           0         ['pool1_pad[0][0]']           
                                                                                                  
 conv2_block1_1_conv (Conv2  (None, 12, 12, 64)           4160      ['pool1_pool[0][0]']          
 D)                                                                                               
                                                                                                  
 conv2_block1_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block1_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block1_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block1_0_conv (Conv2  (None, 12, 12, 256)          16640     ['pool1_pool[0][0]']          
 D)                                                                                               
                                                                                                  
 conv2_block1_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block1_0_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block1_add (Add)      (None, 12, 12, 256)          0         ['conv2_block1_0_bn[0][0]',   
                                                                     'conv2_block1_3_bn[0][0]']   
                                                                                                  
 conv2_block1_out (Activati  (None, 12, 12, 256)          0         ['conv2_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv2_block2_1_conv (Conv2  (None, 12, 12, 64)           16448     ['conv2_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv2_block2_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block2_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block2_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block2_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block2_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block2_add (Add)      (None, 12, 12, 256)          0         ['conv2_block1_out[0][0]',    
                                                                     'conv2_block2_3_bn[0][0]']   
                                                                                                  
 conv2_block2_out (Activati  (None, 12, 12, 256)          0         ['conv2_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv2_block3_1_conv (Conv2  (None, 12, 12, 64)           16448     ['conv2_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv2_block3_1_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_1_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block3_2_conv (Conv2  (None, 12, 12, 64)           36928     ['conv2_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block3_2_bn (BatchNo  (None, 12, 12, 64)           256       ['conv2_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_2_relu (Activ  (None, 12, 12, 64)           0         ['conv2_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv2_block3_3_conv (Conv2  (None, 12, 12, 256)          16640     ['conv2_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv2_block3_3_bn (BatchNo  (None, 12, 12, 256)          1024      ['conv2_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv2_block3_add (Add)      (None, 12, 12, 256)          0         ['conv2_block2_out[0][0]',    
                                                                     'conv2_block3_3_bn[0][0]']   
                                                                                                  
 conv2_block3_out (Activati  (None, 12, 12, 256)          0         ['conv2_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block1_1_conv (Conv2  (None, 6, 6, 128)            32896     ['conv2_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block1_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block1_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block1_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block1_0_conv (Conv2  (None, 6, 6, 512)            131584    ['conv2_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block1_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block1_0_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block1_add (Add)      (None, 6, 6, 512)            0         ['conv3_block1_0_bn[0][0]',   
                                                                     'conv3_block1_3_bn[0][0]']   
                                                                                                  
 conv3_block1_out (Activati  (None, 6, 6, 512)            0         ['conv3_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block2_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block2_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block2_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block2_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block2_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block2_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block2_add (Add)      (None, 6, 6, 512)            0         ['conv3_block1_out[0][0]',    
                                                                     'conv3_block2_3_bn[0][0]']   
                                                                                                  
 conv3_block2_out (Activati  (None, 6, 6, 512)            0         ['conv3_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block3_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block3_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block3_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block3_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block3_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block3_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block3_add (Add)      (None, 6, 6, 512)            0         ['conv3_block2_out[0][0]',    
                                                                     'conv3_block3_3_bn[0][0]']   
                                                                                                  
 conv3_block3_out (Activati  (None, 6, 6, 512)            0         ['conv3_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv3_block4_1_conv (Conv2  (None, 6, 6, 128)            65664     ['conv3_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv3_block4_1_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block4_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_1_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block4_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block4_2_conv (Conv2  (None, 6, 6, 128)            147584    ['conv3_block4_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block4_2_bn (BatchNo  (None, 6, 6, 128)            512       ['conv3_block4_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_2_relu (Activ  (None, 6, 6, 128)            0         ['conv3_block4_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv3_block4_3_conv (Conv2  (None, 6, 6, 512)            66048     ['conv3_block4_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv3_block4_3_bn (BatchNo  (None, 6, 6, 512)            2048      ['conv3_block4_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv3_block4_add (Add)      (None, 6, 6, 512)            0         ['conv3_block3_out[0][0]',    
                                                                     'conv3_block4_3_bn[0][0]']   
                                                                                                  
 conv3_block4_out (Activati  (None, 6, 6, 512)            0         ['conv3_block4_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block1_1_conv (Conv2  (None, 3, 3, 256)            131328    ['conv3_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block1_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block1_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block1_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block1_0_conv (Conv2  (None, 3, 3, 1024)           525312    ['conv3_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block1_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block1_0_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block1_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block1_0_bn[0][0]',   
                                                                     'conv4_block1_3_bn[0][0]']   
                                                                                                  
 conv4_block1_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block2_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block2_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block2_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block2_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block2_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block2_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block2_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block1_out[0][0]',    
                                                                     'conv4_block2_3_bn[0][0]']   
                                                                                                  
 conv4_block2_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block3_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block3_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block3_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block3_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block3_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block3_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block3_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block2_out[0][0]',    
                                                                     'conv4_block3_3_bn[0][0]']   
                                                                                                  
 conv4_block3_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block4_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block3_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block4_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block4_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block4_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block4_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block4_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block4_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block4_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block4_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block4_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block4_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block4_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block4_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block4_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block3_out[0][0]',    
                                                                     'conv4_block4_3_bn[0][0]']   
                                                                                                  
 conv4_block4_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block4_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block5_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block4_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block5_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block5_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block5_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block5_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block5_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block5_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block5_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block5_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block5_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block5_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block5_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block5_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block5_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block4_out[0][0]',    
                                                                     'conv4_block5_3_bn[0][0]']   
                                                                                                  
 conv4_block5_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block5_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block6_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block5_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block6_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block6_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block6_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block6_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block6_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block6_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block6_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block6_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block6_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block6_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block6_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block6_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block6_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block5_out[0][0]',    
                                                                     'conv4_block6_3_bn[0][0]']   
                                                                                                  
 conv4_block6_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block6_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block7_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block6_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block7_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block7_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block7_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block7_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block7_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block7_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block7_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block7_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block7_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block7_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block7_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block7_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block7_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block7_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block7_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block6_out[0][0]',    
                                                                     'conv4_block7_3_bn[0][0]']   
                                                                                                  
 conv4_block7_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block7_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block8_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block7_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block8_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block8_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block8_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block8_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block8_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block8_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block8_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block8_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block8_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block8_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block8_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block8_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block8_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block8_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block8_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block7_out[0][0]',    
                                                                     'conv4_block8_3_bn[0][0]']   
                                                                                                  
 conv4_block8_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block8_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block9_1_conv (Conv2  (None, 3, 3, 256)            262400    ['conv4_block8_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv4_block9_1_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block9_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block9_1_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block9_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block9_2_conv (Conv2  (None, 3, 3, 256)            590080    ['conv4_block9_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block9_2_bn (BatchNo  (None, 3, 3, 256)            1024      ['conv4_block9_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block9_2_relu (Activ  (None, 3, 3, 256)            0         ['conv4_block9_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv4_block9_3_conv (Conv2  (None, 3, 3, 1024)           263168    ['conv4_block9_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv4_block9_3_bn (BatchNo  (None, 3, 3, 1024)           4096      ['conv4_block9_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv4_block9_add (Add)      (None, 3, 3, 1024)           0         ['conv4_block8_out[0][0]',    
                                                                     'conv4_block9_3_bn[0][0]']   
                                                                                                  
 conv4_block9_out (Activati  (None, 3, 3, 1024)           0         ['conv4_block9_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv4_block10_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block9_out[0][0]']    
 2D)                                                                                              
                                                                                                  
 conv4_block10_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block10_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block10_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block10_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block10_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block10_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block10_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block10_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block10_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block10_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block10_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block10_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block10_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block10_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block10_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block9_out[0][0]',    
                                                                     'conv4_block10_3_bn[0][0]']  
                                                                                                  
 conv4_block10_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block10_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block11_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block10_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block11_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block11_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block11_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block11_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block11_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block11_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block11_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block11_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block11_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block11_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block11_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block11_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block11_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block11_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block11_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block10_out[0][0]',   
                                                                     'conv4_block11_3_bn[0][0]']  
                                                                                                  
 conv4_block11_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block11_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block12_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block11_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block12_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block12_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block12_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block12_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block12_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block12_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block12_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block12_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block12_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block12_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block12_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block12_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block12_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block12_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block12_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block11_out[0][0]',   
                                                                     'conv4_block12_3_bn[0][0]']  
                                                                                                  
 conv4_block12_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block12_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block13_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block12_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block13_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block13_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block13_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block13_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block13_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block13_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block13_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block13_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block13_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block13_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block13_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block13_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block13_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block13_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block13_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block12_out[0][0]',   
                                                                     'conv4_block13_3_bn[0][0]']  
                                                                                                  
 conv4_block13_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block13_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block14_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block13_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block14_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block14_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block14_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block14_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block14_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block14_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block14_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block14_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block14_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block14_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block14_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block14_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block14_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block14_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block14_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block13_out[0][0]',   
                                                                     'conv4_block14_3_bn[0][0]']  
                                                                                                  
 conv4_block14_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block14_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block15_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block14_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block15_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block15_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block15_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block15_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block15_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block15_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block15_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block15_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block15_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block15_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block15_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block15_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block15_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block15_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block15_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block14_out[0][0]',   
                                                                     'conv4_block15_3_bn[0][0]']  
                                                                                                  
 conv4_block15_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block15_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block16_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block15_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block16_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block16_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block16_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block16_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block16_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block16_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block16_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block16_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block16_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block16_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block16_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block16_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block16_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block16_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block16_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block15_out[0][0]',   
                                                                     'conv4_block16_3_bn[0][0]']  
                                                                                                  
 conv4_block16_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block16_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block17_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block16_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block17_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block17_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block17_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block17_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block17_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block17_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block17_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block17_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block17_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block17_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block17_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block17_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block17_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block17_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block17_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block16_out[0][0]',   
                                                                     'conv4_block17_3_bn[0][0]']  
                                                                                                  
 conv4_block17_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block17_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block18_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block17_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block18_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block18_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block18_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block18_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block18_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block18_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block18_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block18_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block18_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block18_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block18_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block18_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block18_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block18_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block18_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block17_out[0][0]',   
                                                                     'conv4_block18_3_bn[0][0]']  
                                                                                                  
 conv4_block18_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block18_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block19_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block18_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block19_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block19_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block19_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block19_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block19_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block19_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block19_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block19_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block19_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block19_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block19_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block19_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block19_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block19_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block19_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block18_out[0][0]',   
                                                                     'conv4_block19_3_bn[0][0]']  
                                                                                                  
 conv4_block19_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block19_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block20_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block19_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block20_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block20_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block20_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block20_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block20_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block20_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block20_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block20_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block20_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block20_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block20_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block20_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block20_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block20_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block20_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block19_out[0][0]',   
                                                                     'conv4_block20_3_bn[0][0]']  
                                                                                                  
 conv4_block20_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block20_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block21_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block20_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block21_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block21_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block21_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block21_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block21_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block21_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block21_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block21_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block21_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block21_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block21_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block21_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block21_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block21_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block21_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block20_out[0][0]',   
                                                                     'conv4_block21_3_bn[0][0]']  
                                                                                                  
 conv4_block21_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block21_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block22_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block21_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block22_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block22_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block22_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block22_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block22_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block22_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block22_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block22_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block22_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block22_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block22_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block22_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block22_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block22_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block22_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block21_out[0][0]',   
                                                                     'conv4_block22_3_bn[0][0]']  
                                                                                                  
 conv4_block22_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block22_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv4_block23_1_conv (Conv  (None, 3, 3, 256)            262400    ['conv4_block22_out[0][0]']   
 2D)                                                                                              
                                                                                                  
 conv4_block23_1_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block23_1_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block23_1_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block23_1_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block23_2_conv (Conv  (None, 3, 3, 256)            590080    ['conv4_block23_1_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block23_2_bn (BatchN  (None, 3, 3, 256)            1024      ['conv4_block23_2_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block23_2_relu (Acti  (None, 3, 3, 256)            0         ['conv4_block23_2_bn[0][0]']  
 vation)                                                                                          
                                                                                                  
 conv4_block23_3_conv (Conv  (None, 3, 3, 1024)           263168    ['conv4_block23_2_relu[0][0]']
 2D)                                                                                              
                                                                                                  
 conv4_block23_3_bn (BatchN  (None, 3, 3, 1024)           4096      ['conv4_block23_3_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 conv4_block23_add (Add)     (None, 3, 3, 1024)           0         ['conv4_block22_out[0][0]',   
                                                                     'conv4_block23_3_bn[0][0]']  
                                                                                                  
 conv4_block23_out (Activat  (None, 3, 3, 1024)           0         ['conv4_block23_add[0][0]']   
 ion)                                                                                             
                                                                                                  
 conv5_block1_1_conv (Conv2  (None, 2, 2, 512)            524800    ['conv4_block23_out[0][0]']   
 D)                                                                                               
                                                                                                  
 conv5_block1_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block1_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block1_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block1_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block1_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block1_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block1_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block1_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block1_0_conv (Conv2  (None, 2, 2, 2048)           2099200   ['conv4_block23_out[0][0]']   
 D)                                                                                               
                                                                                                  
 conv5_block1_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block1_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block1_0_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block1_0_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block1_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block1_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block1_0_bn[0][0]',   
                                                                     'conv5_block1_3_bn[0][0]']   
                                                                                                  
 conv5_block1_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block1_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv5_block2_1_conv (Conv2  (None, 2, 2, 512)            1049088   ['conv5_block1_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block2_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block2_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block2_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block2_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block2_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block2_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block2_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block2_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block2_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block2_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block2_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block2_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block2_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block1_out[0][0]',    
                                                                     'conv5_block2_3_bn[0][0]']   
                                                                                                  
 conv5_block2_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block2_add[0][0]']    
 on)                                                                                              
                                                                                                  
 conv5_block3_1_conv (Conv2  (None, 2, 2, 512)            1049088   ['conv5_block2_out[0][0]']    
 D)                                                                                               
                                                                                                  
 conv5_block3_1_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block3_1_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_1_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block3_1_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block3_2_conv (Conv2  (None, 2, 2, 512)            2359808   ['conv5_block3_1_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block3_2_bn (BatchNo  (None, 2, 2, 512)            2048      ['conv5_block3_2_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_2_relu (Activ  (None, 2, 2, 512)            0         ['conv5_block3_2_bn[0][0]']   
 ation)                                                                                           
                                                                                                  
 conv5_block3_3_conv (Conv2  (None, 2, 2, 2048)           1050624   ['conv5_block3_2_relu[0][0]'] 
 D)                                                                                               
                                                                                                  
 conv5_block3_3_bn (BatchNo  (None, 2, 2, 2048)           8192      ['conv5_block3_3_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 conv5_block3_add (Add)      (None, 2, 2, 2048)           0         ['conv5_block2_out[0][0]',    
                                                                     'conv5_block3_3_bn[0][0]']   
                                                                                                  
 conv5_block3_out (Activati  (None, 2, 2, 2048)           0         ['conv5_block3_add[0][0]']    
 on)                                                                                              
                                                                                                  
==================================================================================================
Total params: 42658176 (162.73 MB)
Trainable params: 42552832 (162.33 MB)
Non-trainable params: 105344 (411.50 KB)
__________________________________________________________________________________________________
In [ ]:
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"

# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)

# Prepare the data generators
train_generator = datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

validation_generator = datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

test_generator = datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    shuffle=False)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Compiling and Training the Model¶

In [ ]:
np.random.seed(42)
In [ ]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D
from tensorflow.keras.layers import Flatten, Dense, Dropout, BatchNormalization
from tensorflow.keras.optimizers import AdamW, SGD, Adamax
# Load ResNet50 model pre-trained on ImageNet without the top layer
base_model = ResNet50(weights='imagenet', include_top=False, input_shape=(48, 48, 3))

# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.4)(x)

predictions = Dense(4, activation='softmax')(x)


resnet_model2 = Model(inputs=base_model.input, outputs=predictions)
# Compile the model
resnet_model2.compile(optimizer=Adamax(learning_rate=0.00025), loss='categorical_crossentropy', metrics=['accuracy'])
In [ ]:
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)


history = resnet_model2.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // train_generator.batch_size,
    epochs=20,
    validation_data=validation_generator,
    validation_steps=validation_generator.samples // validation_generator.batch_size,
    class_weight=class_weights_adjusted,
    callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/20
472/472 [==============================] - ETA: 0s - loss: 1.2432 - accuracy: 0.4541
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
472/472 [==============================] - 53s 62ms/step - loss: 1.2432 - accuracy: 0.4541 - val_loss: 148.1311 - val_accuracy: 0.2230 - lr: 2.5000e-04
Epoch 2/20
472/472 [==============================] - 27s 57ms/step - loss: 0.8767 - accuracy: 0.6330 - val_loss: 1.2080 - val_accuracy: 0.4980 - lr: 2.5000e-04
Epoch 3/20
472/472 [==============================] - 27s 57ms/step - loss: 0.6773 - accuracy: 0.7215 - val_loss: 0.8134 - val_accuracy: 0.6853 - lr: 2.5000e-04
Epoch 4/20
472/472 [==============================] - 26s 55ms/step - loss: 0.5270 - accuracy: 0.7853 - val_loss: 0.8374 - val_accuracy: 0.6808 - lr: 2.5000e-04
Epoch 5/20
472/472 [==============================] - 27s 58ms/step - loss: 0.3790 - accuracy: 0.8498 - val_loss: 0.8913 - val_accuracy: 0.7121 - lr: 2.5000e-04
Epoch 6/20
472/472 [==============================] - 26s 55ms/step - loss: 0.2578 - accuracy: 0.9021 - val_loss: 1.2126 - val_accuracy: 0.6875 - lr: 2.5000e-04
Epoch 7/20
472/472 [==============================] - 27s 57ms/step - loss: 0.1923 - accuracy: 0.9292 - val_loss: 0.9888 - val_accuracy: 0.7210 - lr: 2.5000e-04
Epoch 8/20
472/472 [==============================] - 26s 54ms/step - loss: 0.1055 - accuracy: 0.9625 - val_loss: 1.2811 - val_accuracy: 0.7185 - lr: 2.5000e-04
Epoch 9/20
472/472 [==============================] - 27s 57ms/step - loss: 0.0546 - accuracy: 0.9821 - val_loss: 1.2760 - val_accuracy: 0.7306 - lr: 5.0000e-05
Epoch 10/20
472/472 [==============================] - 27s 57ms/step - loss: 0.0351 - accuracy: 0.9884 - val_loss: 1.3550 - val_accuracy: 0.7310 - lr: 5.0000e-05

Evaluating the ResNet Model¶

In [ ]:
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(test_generator.classes, np.argmax(resnet_model2.predict(test_generator), axis=1))

# Confusion matrix
print("Confusion Matrix:")
print(cm)

# Classification report
# Plot the training and validation accuracy values
plt.figure(figsize=(15, 5))
plt.subplot(1, 3, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model_resnet Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 3, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model_resnet Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 3, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for Resnet Model")
plt.show()

predicted_labels = np.argmax(resnet_model2.predict(test_generator), axis=1)

# Get the true labels for the test set
true_labels = test_generator.classes

# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)

# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)

# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 1s 20ms/step
Confusion Matrix:
[[26  2  2  2]
 [ 2 25  5  0]
 [ 1 12 17  2]
 [ 0  3  1 28]]
4/4 [==============================] - 0s 22ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
              precision    recall  f1-score   support

           0       0.90      0.81      0.85        32
           1       0.60      0.78      0.68        32
           2       0.68      0.53      0.60        32
           3       0.88      0.88      0.88        32

    accuracy                           0.75       128
   macro avg       0.76      0.75      0.75       128
weighted avg       0.76      0.75      0.75       128

In [ ]:
# Evaluate the model on the test set
val_loss, val_accuracy = resnet_model2.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)

print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')

The ResNet50-based custom model, adapted for classifying four emotional states, achieved an overall accuracy of 75%. The model's precision and recall suggest it performs best in recognizing 'happy' and 'surprise' emotions, with scores indicating high reliability in these predictions. However, it shows a need for improvement in accurately classifying 'neutral' and 'sad' states, as evidenced by lower precision for 'neutral' and lower recall for 'sad'. The confusion matrix details the model's performance across all classes, highlighting specific areas where misclassifications occur, particularly in distinguishing 'neutral' from 'sad'. Optimizations could focus on enhancing feature extraction and addressing class imbalance to improve accuracy and balance across all emotion categories.



EfficientNet Model¶

Model Building¶

  • Import EfficientNet upto the layer of your choice and add Fully Connected layers on top of it.
In [112]:
np.random.seed(42)
In [113]:
import tensorflow as tf
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications.efficientnet import EfficientNetB0, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D
from tensorflow.keras import regularizers
In [114]:
import tensorflow as tf
import tensorflow.keras.applications as ap
EfficientNet = ap.EfficientNetV2B2(include_top=False,weights="imagenet", input_shape= (48, 48, 3))

EfficientNet.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/efficientnet_v2/efficientnetv2-b2_notop.h5
35839040/35839040 [==============================] - 3s 0us/step
Model: "efficientnetv2-b2"
__________________________________________________________________________________________________
 Layer (type)                Output Shape                 Param #   Connected to                  
==================================================================================================
 input_1 (InputLayer)        [(None, 48, 48, 3)]          0         []                            
                                                                                                  
 rescaling (Rescaling)       (None, 48, 48, 3)            0         ['input_1[0][0]']             
                                                                                                  
 normalization (Normalizati  (None, 48, 48, 3)            0         ['rescaling[0][0]']           
 on)                                                                                              
                                                                                                  
 stem_conv (Conv2D)          (None, 24, 24, 32)           864       ['normalization[0][0]']       
                                                                                                  
 stem_bn (BatchNormalizatio  (None, 24, 24, 32)           128       ['stem_conv[0][0]']           
 n)                                                                                               
                                                                                                  
 stem_activation (Activatio  (None, 24, 24, 32)           0         ['stem_bn[0][0]']             
 n)                                                                                               
                                                                                                  
 block1a_project_conv (Conv  (None, 24, 24, 16)           4608      ['stem_activation[0][0]']     
 2D)                                                                                              
                                                                                                  
 block1a_project_bn (BatchN  (None, 24, 24, 16)           64        ['block1a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block1a_project_activation  (None, 24, 24, 16)           0         ['block1a_project_bn[0][0]']  
  (Activation)                                                                                    
                                                                                                  
 block1b_project_conv (Conv  (None, 24, 24, 16)           2304      ['block1a_project_activation[0
 2D)                                                                ][0]']                        
                                                                                                  
 block1b_project_bn (BatchN  (None, 24, 24, 16)           64        ['block1b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block1b_project_activation  (None, 24, 24, 16)           0         ['block1b_project_bn[0][0]']  
  (Activation)                                                                                    
                                                                                                  
 block1b_drop (Dropout)      (None, 24, 24, 16)           0         ['block1b_project_activation[0
                                                                    ][0]']                        
                                                                                                  
 block1b_add (Add)           (None, 24, 24, 16)           0         ['block1b_drop[0][0]',        
                                                                     'block1a_project_activation[0
                                                                    ][0]']                        
                                                                                                  
 block2a_expand_conv (Conv2  (None, 12, 12, 64)           9216      ['block1b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block2a_expand_bn (BatchNo  (None, 12, 12, 64)           256       ['block2a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2a_expand_activation   (None, 12, 12, 64)           0         ['block2a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2a_project_conv (Conv  (None, 12, 12, 32)           2048      ['block2a_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2a_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2b_expand_conv (Conv2  (None, 12, 12, 128)          36864     ['block2a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block2b_expand_bn (BatchNo  (None, 12, 12, 128)          512       ['block2b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2b_expand_activation   (None, 12, 12, 128)          0         ['block2b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2b_project_conv (Conv  (None, 12, 12, 32)           4096      ['block2b_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2b_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2b_drop (Dropout)      (None, 12, 12, 32)           0         ['block2b_project_bn[0][0]']  
                                                                                                  
 block2b_add (Add)           (None, 12, 12, 32)           0         ['block2b_drop[0][0]',        
                                                                     'block2a_project_bn[0][0]']  
                                                                                                  
 block2c_expand_conv (Conv2  (None, 12, 12, 128)          36864     ['block2b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block2c_expand_bn (BatchNo  (None, 12, 12, 128)          512       ['block2c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block2c_expand_activation   (None, 12, 12, 128)          0         ['block2c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block2c_project_conv (Conv  (None, 12, 12, 32)           4096      ['block2c_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block2c_project_bn (BatchN  (None, 12, 12, 32)           128       ['block2c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block2c_drop (Dropout)      (None, 12, 12, 32)           0         ['block2c_project_bn[0][0]']  
                                                                                                  
 block2c_add (Add)           (None, 12, 12, 32)           0         ['block2c_drop[0][0]',        
                                                                     'block2b_add[0][0]']         
                                                                                                  
 block3a_expand_conv (Conv2  (None, 6, 6, 128)            36864     ['block2c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block3a_expand_bn (BatchNo  (None, 6, 6, 128)            512       ['block3a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3a_expand_activation   (None, 6, 6, 128)            0         ['block3a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3a_project_conv (Conv  (None, 6, 6, 56)             7168      ['block3a_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3a_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3b_expand_conv (Conv2  (None, 6, 6, 224)            112896    ['block3a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block3b_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block3b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3b_expand_activation   (None, 6, 6, 224)            0         ['block3b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3b_project_conv (Conv  (None, 6, 6, 56)             12544     ['block3b_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3b_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3b_drop (Dropout)      (None, 6, 6, 56)             0         ['block3b_project_bn[0][0]']  
                                                                                                  
 block3b_add (Add)           (None, 6, 6, 56)             0         ['block3b_drop[0][0]',        
                                                                     'block3a_project_bn[0][0]']  
                                                                                                  
 block3c_expand_conv (Conv2  (None, 6, 6, 224)            112896    ['block3b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block3c_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block3c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block3c_expand_activation   (None, 6, 6, 224)            0         ['block3c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block3c_project_conv (Conv  (None, 6, 6, 56)             12544     ['block3c_expand_activation[0]
 2D)                                                                [0]']                         
                                                                                                  
 block3c_project_bn (BatchN  (None, 6, 6, 56)             224       ['block3c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block3c_drop (Dropout)      (None, 6, 6, 56)             0         ['block3c_project_bn[0][0]']  
                                                                                                  
 block3c_add (Add)           (None, 6, 6, 56)             0         ['block3c_drop[0][0]',        
                                                                     'block3b_add[0][0]']         
                                                                                                  
 block4a_expand_conv (Conv2  (None, 6, 6, 224)            12544     ['block3c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4a_expand_bn (BatchNo  (None, 6, 6, 224)            896       ['block4a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4a_expand_activation   (None, 6, 6, 224)            0         ['block4a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4a_dwconv2 (Depthwise  (None, 3, 3, 224)            2016      ['block4a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4a_bn (BatchNormaliza  (None, 3, 3, 224)            896       ['block4a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4a_activation (Activa  (None, 3, 3, 224)            0         ['block4a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4a_se_squeeze (Global  (None, 224)                  0         ['block4a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4a_se_reshape (Reshap  (None, 1, 1, 224)            0         ['block4a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4a_se_reduce (Conv2D)  (None, 1, 1, 14)             3150      ['block4a_se_reshape[0][0]']  
                                                                                                  
 block4a_se_expand (Conv2D)  (None, 1, 1, 224)            3360      ['block4a_se_reduce[0][0]']   
                                                                                                  
 block4a_se_excite (Multipl  (None, 3, 3, 224)            0         ['block4a_activation[0][0]',  
 y)                                                                  'block4a_se_expand[0][0]']   
                                                                                                  
 block4a_project_conv (Conv  (None, 3, 3, 104)            23296     ['block4a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4a_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4b_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block4b_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4b_expand_activation   (None, 3, 3, 416)            0         ['block4b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4b_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4b_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4b_activation (Activa  (None, 3, 3, 416)            0         ['block4b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4b_se_squeeze (Global  (None, 416)                  0         ['block4b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4b_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4b_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4b_se_reshape[0][0]']  
                                                                                                  
 block4b_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4b_se_reduce[0][0]']   
                                                                                                  
 block4b_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4b_activation[0][0]',  
 y)                                                                  'block4b_se_expand[0][0]']   
                                                                                                  
 block4b_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4b_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4b_drop (Dropout)      (None, 3, 3, 104)            0         ['block4b_project_bn[0][0]']  
                                                                                                  
 block4b_add (Add)           (None, 3, 3, 104)            0         ['block4b_drop[0][0]',        
                                                                     'block4a_project_bn[0][0]']  
                                                                                                  
 block4c_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4c_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4c_expand_activation   (None, 3, 3, 416)            0         ['block4c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4c_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4c_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4c_activation (Activa  (None, 3, 3, 416)            0         ['block4c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4c_se_squeeze (Global  (None, 416)                  0         ['block4c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4c_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4c_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4c_se_reshape[0][0]']  
                                                                                                  
 block4c_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4c_se_reduce[0][0]']   
                                                                                                  
 block4c_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4c_activation[0][0]',  
 y)                                                                  'block4c_se_expand[0][0]']   
                                                                                                  
 block4c_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4c_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4c_drop (Dropout)      (None, 3, 3, 104)            0         ['block4c_project_bn[0][0]']  
                                                                                                  
 block4c_add (Add)           (None, 3, 3, 104)            0         ['block4c_drop[0][0]',        
                                                                     'block4b_add[0][0]']         
                                                                                                  
 block4d_expand_conv (Conv2  (None, 3, 3, 416)            43264     ['block4c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block4d_expand_bn (BatchNo  (None, 3, 3, 416)            1664      ['block4d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block4d_expand_activation   (None, 3, 3, 416)            0         ['block4d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block4d_dwconv2 (Depthwise  (None, 3, 3, 416)            3744      ['block4d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block4d_bn (BatchNormaliza  (None, 3, 3, 416)            1664      ['block4d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block4d_activation (Activa  (None, 3, 3, 416)            0         ['block4d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block4d_se_squeeze (Global  (None, 416)                  0         ['block4d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block4d_se_reshape (Reshap  (None, 1, 1, 416)            0         ['block4d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block4d_se_reduce (Conv2D)  (None, 1, 1, 26)             10842     ['block4d_se_reshape[0][0]']  
                                                                                                  
 block4d_se_expand (Conv2D)  (None, 1, 1, 416)            11232     ['block4d_se_reduce[0][0]']   
                                                                                                  
 block4d_se_excite (Multipl  (None, 3, 3, 416)            0         ['block4d_activation[0][0]',  
 y)                                                                  'block4d_se_expand[0][0]']   
                                                                                                  
 block4d_project_conv (Conv  (None, 3, 3, 104)            43264     ['block4d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block4d_project_bn (BatchN  (None, 3, 3, 104)            416       ['block4d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block4d_drop (Dropout)      (None, 3, 3, 104)            0         ['block4d_project_bn[0][0]']  
                                                                                                  
 block4d_add (Add)           (None, 3, 3, 104)            0         ['block4d_drop[0][0]',        
                                                                     'block4c_add[0][0]']         
                                                                                                  
 block5a_expand_conv (Conv2  (None, 3, 3, 624)            64896     ['block4d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5a_expand_bn (BatchNo  (None, 3, 3, 624)            2496      ['block5a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5a_expand_activation   (None, 3, 3, 624)            0         ['block5a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5a_dwconv2 (Depthwise  (None, 3, 3, 624)            5616      ['block5a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5a_bn (BatchNormaliza  (None, 3, 3, 624)            2496      ['block5a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5a_activation (Activa  (None, 3, 3, 624)            0         ['block5a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5a_se_squeeze (Global  (None, 624)                  0         ['block5a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5a_se_reshape (Reshap  (None, 1, 1, 624)            0         ['block5a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5a_se_reduce (Conv2D)  (None, 1, 1, 26)             16250     ['block5a_se_reshape[0][0]']  
                                                                                                  
 block5a_se_expand (Conv2D)  (None, 1, 1, 624)            16848     ['block5a_se_reduce[0][0]']   
                                                                                                  
 block5a_se_excite (Multipl  (None, 3, 3, 624)            0         ['block5a_activation[0][0]',  
 y)                                                                  'block5a_se_expand[0][0]']   
                                                                                                  
 block5a_project_conv (Conv  (None, 3, 3, 120)            74880     ['block5a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5a_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5b_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block5b_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5b_expand_activation   (None, 3, 3, 720)            0         ['block5b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5b_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5b_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5b_activation (Activa  (None, 3, 3, 720)            0         ['block5b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5b_se_squeeze (Global  (None, 720)                  0         ['block5b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5b_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5b_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5b_se_reshape[0][0]']  
                                                                                                  
 block5b_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5b_se_reduce[0][0]']   
                                                                                                  
 block5b_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5b_activation[0][0]',  
 y)                                                                  'block5b_se_expand[0][0]']   
                                                                                                  
 block5b_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5b_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5b_drop (Dropout)      (None, 3, 3, 120)            0         ['block5b_project_bn[0][0]']  
                                                                                                  
 block5b_add (Add)           (None, 3, 3, 120)            0         ['block5b_drop[0][0]',        
                                                                     'block5a_project_bn[0][0]']  
                                                                                                  
 block5c_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5c_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5c_expand_activation   (None, 3, 3, 720)            0         ['block5c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5c_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5c_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5c_activation (Activa  (None, 3, 3, 720)            0         ['block5c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5c_se_squeeze (Global  (None, 720)                  0         ['block5c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5c_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5c_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5c_se_reshape[0][0]']  
                                                                                                  
 block5c_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5c_se_reduce[0][0]']   
                                                                                                  
 block5c_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5c_activation[0][0]',  
 y)                                                                  'block5c_se_expand[0][0]']   
                                                                                                  
 block5c_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5c_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5c_drop (Dropout)      (None, 3, 3, 120)            0         ['block5c_project_bn[0][0]']  
                                                                                                  
 block5c_add (Add)           (None, 3, 3, 120)            0         ['block5c_drop[0][0]',        
                                                                     'block5b_add[0][0]']         
                                                                                                  
 block5d_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5d_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5d_expand_activation   (None, 3, 3, 720)            0         ['block5d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5d_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5d_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5d_activation (Activa  (None, 3, 3, 720)            0         ['block5d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5d_se_squeeze (Global  (None, 720)                  0         ['block5d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5d_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5d_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5d_se_reshape[0][0]']  
                                                                                                  
 block5d_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5d_se_reduce[0][0]']   
                                                                                                  
 block5d_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5d_activation[0][0]',  
 y)                                                                  'block5d_se_expand[0][0]']   
                                                                                                  
 block5d_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5d_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5d_drop (Dropout)      (None, 3, 3, 120)            0         ['block5d_project_bn[0][0]']  
                                                                                                  
 block5d_add (Add)           (None, 3, 3, 120)            0         ['block5d_drop[0][0]',        
                                                                     'block5c_add[0][0]']         
                                                                                                  
 block5e_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5e_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5e_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5e_expand_activation   (None, 3, 3, 720)            0         ['block5e_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5e_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5e_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5e_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5e_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5e_activation (Activa  (None, 3, 3, 720)            0         ['block5e_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5e_se_squeeze (Global  (None, 720)                  0         ['block5e_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5e_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5e_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5e_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5e_se_reshape[0][0]']  
                                                                                                  
 block5e_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5e_se_reduce[0][0]']   
                                                                                                  
 block5e_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5e_activation[0][0]',  
 y)                                                                  'block5e_se_expand[0][0]']   
                                                                                                  
 block5e_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5e_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5e_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5e_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5e_drop (Dropout)      (None, 3, 3, 120)            0         ['block5e_project_bn[0][0]']  
                                                                                                  
 block5e_add (Add)           (None, 3, 3, 120)            0         ['block5e_drop[0][0]',        
                                                                     'block5d_add[0][0]']         
                                                                                                  
 block5f_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5e_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block5f_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block5f_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block5f_expand_activation   (None, 3, 3, 720)            0         ['block5f_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block5f_dwconv2 (Depthwise  (None, 3, 3, 720)            6480      ['block5f_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block5f_bn (BatchNormaliza  (None, 3, 3, 720)            2880      ['block5f_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block5f_activation (Activa  (None, 3, 3, 720)            0         ['block5f_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block5f_se_squeeze (Global  (None, 720)                  0         ['block5f_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block5f_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block5f_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block5f_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block5f_se_reshape[0][0]']  
                                                                                                  
 block5f_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block5f_se_reduce[0][0]']   
                                                                                                  
 block5f_se_excite (Multipl  (None, 3, 3, 720)            0         ['block5f_activation[0][0]',  
 y)                                                                  'block5f_se_expand[0][0]']   
                                                                                                  
 block5f_project_conv (Conv  (None, 3, 3, 120)            86400     ['block5f_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block5f_project_bn (BatchN  (None, 3, 3, 120)            480       ['block5f_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block5f_drop (Dropout)      (None, 3, 3, 120)            0         ['block5f_project_bn[0][0]']  
                                                                                                  
 block5f_add (Add)           (None, 3, 3, 120)            0         ['block5f_drop[0][0]',        
                                                                     'block5e_add[0][0]']         
                                                                                                  
 block6a_expand_conv (Conv2  (None, 3, 3, 720)            86400     ['block5f_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6a_expand_bn (BatchNo  (None, 3, 3, 720)            2880      ['block6a_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6a_expand_activation   (None, 3, 3, 720)            0         ['block6a_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6a_dwconv2 (Depthwise  (None, 2, 2, 720)            6480      ['block6a_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6a_bn (BatchNormaliza  (None, 2, 2, 720)            2880      ['block6a_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6a_activation (Activa  (None, 2, 2, 720)            0         ['block6a_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6a_se_squeeze (Global  (None, 720)                  0         ['block6a_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6a_se_reshape (Reshap  (None, 1, 1, 720)            0         ['block6a_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6a_se_reduce (Conv2D)  (None, 1, 1, 30)             21630     ['block6a_se_reshape[0][0]']  
                                                                                                  
 block6a_se_expand (Conv2D)  (None, 1, 1, 720)            22320     ['block6a_se_reduce[0][0]']   
                                                                                                  
 block6a_se_excite (Multipl  (None, 2, 2, 720)            0         ['block6a_activation[0][0]',  
 y)                                                                  'block6a_se_expand[0][0]']   
                                                                                                  
 block6a_project_conv (Conv  (None, 2, 2, 208)            149760    ['block6a_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6a_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6a_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6b_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6a_project_bn[0][0]']  
 D)                                                                                               
                                                                                                  
 block6b_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6b_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6b_expand_activation   (None, 2, 2, 1248)           0         ['block6b_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6b_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6b_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6b_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6b_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6b_activation (Activa  (None, 2, 2, 1248)           0         ['block6b_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6b_se_squeeze (Global  (None, 1248)                 0         ['block6b_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6b_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6b_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6b_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6b_se_reshape[0][0]']  
                                                                                                  
 block6b_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6b_se_reduce[0][0]']   
                                                                                                  
 block6b_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6b_activation[0][0]',  
 y)                                                                  'block6b_se_expand[0][0]']   
                                                                                                  
 block6b_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6b_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6b_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6b_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6b_drop (Dropout)      (None, 2, 2, 208)            0         ['block6b_project_bn[0][0]']  
                                                                                                  
 block6b_add (Add)           (None, 2, 2, 208)            0         ['block6b_drop[0][0]',        
                                                                     'block6a_project_bn[0][0]']  
                                                                                                  
 block6c_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6b_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6c_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6c_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6c_expand_activation   (None, 2, 2, 1248)           0         ['block6c_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6c_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6c_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6c_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6c_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6c_activation (Activa  (None, 2, 2, 1248)           0         ['block6c_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6c_se_squeeze (Global  (None, 1248)                 0         ['block6c_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6c_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6c_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6c_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6c_se_reshape[0][0]']  
                                                                                                  
 block6c_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6c_se_reduce[0][0]']   
                                                                                                  
 block6c_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6c_activation[0][0]',  
 y)                                                                  'block6c_se_expand[0][0]']   
                                                                                                  
 block6c_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6c_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6c_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6c_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6c_drop (Dropout)      (None, 2, 2, 208)            0         ['block6c_project_bn[0][0]']  
                                                                                                  
 block6c_add (Add)           (None, 2, 2, 208)            0         ['block6c_drop[0][0]',        
                                                                     'block6b_add[0][0]']         
                                                                                                  
 block6d_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6c_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6d_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6d_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6d_expand_activation   (None, 2, 2, 1248)           0         ['block6d_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6d_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6d_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6d_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6d_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6d_activation (Activa  (None, 2, 2, 1248)           0         ['block6d_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6d_se_squeeze (Global  (None, 1248)                 0         ['block6d_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6d_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6d_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6d_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6d_se_reshape[0][0]']  
                                                                                                  
 block6d_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6d_se_reduce[0][0]']   
                                                                                                  
 block6d_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6d_activation[0][0]',  
 y)                                                                  'block6d_se_expand[0][0]']   
                                                                                                  
 block6d_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6d_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6d_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6d_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6d_drop (Dropout)      (None, 2, 2, 208)            0         ['block6d_project_bn[0][0]']  
                                                                                                  
 block6d_add (Add)           (None, 2, 2, 208)            0         ['block6d_drop[0][0]',        
                                                                     'block6c_add[0][0]']         
                                                                                                  
 block6e_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6d_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6e_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6e_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6e_expand_activation   (None, 2, 2, 1248)           0         ['block6e_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6e_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6e_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6e_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6e_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6e_activation (Activa  (None, 2, 2, 1248)           0         ['block6e_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6e_se_squeeze (Global  (None, 1248)                 0         ['block6e_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6e_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6e_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6e_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6e_se_reshape[0][0]']  
                                                                                                  
 block6e_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6e_se_reduce[0][0]']   
                                                                                                  
 block6e_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6e_activation[0][0]',  
 y)                                                                  'block6e_se_expand[0][0]']   
                                                                                                  
 block6e_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6e_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6e_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6e_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6e_drop (Dropout)      (None, 2, 2, 208)            0         ['block6e_project_bn[0][0]']  
                                                                                                  
 block6e_add (Add)           (None, 2, 2, 208)            0         ['block6e_drop[0][0]',        
                                                                     'block6d_add[0][0]']         
                                                                                                  
 block6f_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6e_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6f_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6f_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6f_expand_activation   (None, 2, 2, 1248)           0         ['block6f_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6f_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6f_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6f_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6f_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6f_activation (Activa  (None, 2, 2, 1248)           0         ['block6f_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6f_se_squeeze (Global  (None, 1248)                 0         ['block6f_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6f_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6f_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6f_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6f_se_reshape[0][0]']  
                                                                                                  
 block6f_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6f_se_reduce[0][0]']   
                                                                                                  
 block6f_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6f_activation[0][0]',  
 y)                                                                  'block6f_se_expand[0][0]']   
                                                                                                  
 block6f_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6f_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6f_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6f_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6f_drop (Dropout)      (None, 2, 2, 208)            0         ['block6f_project_bn[0][0]']  
                                                                                                  
 block6f_add (Add)           (None, 2, 2, 208)            0         ['block6f_drop[0][0]',        
                                                                     'block6e_add[0][0]']         
                                                                                                  
 block6g_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6f_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6g_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6g_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6g_expand_activation   (None, 2, 2, 1248)           0         ['block6g_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6g_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6g_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6g_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6g_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6g_activation (Activa  (None, 2, 2, 1248)           0         ['block6g_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6g_se_squeeze (Global  (None, 1248)                 0         ['block6g_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6g_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6g_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6g_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6g_se_reshape[0][0]']  
                                                                                                  
 block6g_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6g_se_reduce[0][0]']   
                                                                                                  
 block6g_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6g_activation[0][0]',  
 y)                                                                  'block6g_se_expand[0][0]']   
                                                                                                  
 block6g_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6g_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6g_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6g_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6g_drop (Dropout)      (None, 2, 2, 208)            0         ['block6g_project_bn[0][0]']  
                                                                                                  
 block6g_add (Add)           (None, 2, 2, 208)            0         ['block6g_drop[0][0]',        
                                                                     'block6f_add[0][0]']         
                                                                                                  
 block6h_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6g_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6h_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6h_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6h_expand_activation   (None, 2, 2, 1248)           0         ['block6h_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6h_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6h_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6h_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6h_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6h_activation (Activa  (None, 2, 2, 1248)           0         ['block6h_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6h_se_squeeze (Global  (None, 1248)                 0         ['block6h_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6h_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6h_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6h_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6h_se_reshape[0][0]']  
                                                                                                  
 block6h_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6h_se_reduce[0][0]']   
                                                                                                  
 block6h_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6h_activation[0][0]',  
 y)                                                                  'block6h_se_expand[0][0]']   
                                                                                                  
 block6h_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6h_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6h_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6h_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6h_drop (Dropout)      (None, 2, 2, 208)            0         ['block6h_project_bn[0][0]']  
                                                                                                  
 block6h_add (Add)           (None, 2, 2, 208)            0         ['block6h_drop[0][0]',        
                                                                     'block6g_add[0][0]']         
                                                                                                  
 block6i_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6h_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6i_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6i_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6i_expand_activation   (None, 2, 2, 1248)           0         ['block6i_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6i_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6i_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6i_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6i_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6i_activation (Activa  (None, 2, 2, 1248)           0         ['block6i_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6i_se_squeeze (Global  (None, 1248)                 0         ['block6i_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6i_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6i_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6i_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6i_se_reshape[0][0]']  
                                                                                                  
 block6i_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6i_se_reduce[0][0]']   
                                                                                                  
 block6i_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6i_activation[0][0]',  
 y)                                                                  'block6i_se_expand[0][0]']   
                                                                                                  
 block6i_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6i_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6i_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6i_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6i_drop (Dropout)      (None, 2, 2, 208)            0         ['block6i_project_bn[0][0]']  
                                                                                                  
 block6i_add (Add)           (None, 2, 2, 208)            0         ['block6i_drop[0][0]',        
                                                                     'block6h_add[0][0]']         
                                                                                                  
 block6j_expand_conv (Conv2  (None, 2, 2, 1248)           259584    ['block6i_add[0][0]']         
 D)                                                                                               
                                                                                                  
 block6j_expand_bn (BatchNo  (None, 2, 2, 1248)           4992      ['block6j_expand_conv[0][0]'] 
 rmalization)                                                                                     
                                                                                                  
 block6j_expand_activation   (None, 2, 2, 1248)           0         ['block6j_expand_bn[0][0]']   
 (Activation)                                                                                     
                                                                                                  
 block6j_dwconv2 (Depthwise  (None, 2, 2, 1248)           11232     ['block6j_expand_activation[0]
 Conv2D)                                                            [0]']                         
                                                                                                  
 block6j_bn (BatchNormaliza  (None, 2, 2, 1248)           4992      ['block6j_dwconv2[0][0]']     
 tion)                                                                                            
                                                                                                  
 block6j_activation (Activa  (None, 2, 2, 1248)           0         ['block6j_bn[0][0]']          
 tion)                                                                                            
                                                                                                  
 block6j_se_squeeze (Global  (None, 1248)                 0         ['block6j_activation[0][0]']  
 AveragePooling2D)                                                                                
                                                                                                  
 block6j_se_reshape (Reshap  (None, 1, 1, 1248)           0         ['block6j_se_squeeze[0][0]']  
 e)                                                                                               
                                                                                                  
 block6j_se_reduce (Conv2D)  (None, 1, 1, 52)             64948     ['block6j_se_reshape[0][0]']  
                                                                                                  
 block6j_se_expand (Conv2D)  (None, 1, 1, 1248)           66144     ['block6j_se_reduce[0][0]']   
                                                                                                  
 block6j_se_excite (Multipl  (None, 2, 2, 1248)           0         ['block6j_activation[0][0]',  
 y)                                                                  'block6j_se_expand[0][0]']   
                                                                                                  
 block6j_project_conv (Conv  (None, 2, 2, 208)            259584    ['block6j_se_excite[0][0]']   
 2D)                                                                                              
                                                                                                  
 block6j_project_bn (BatchN  (None, 2, 2, 208)            832       ['block6j_project_conv[0][0]']
 ormalization)                                                                                    
                                                                                                  
 block6j_drop (Dropout)      (None, 2, 2, 208)            0         ['block6j_project_bn[0][0]']  
                                                                                                  
 block6j_add (Add)           (None, 2, 2, 208)            0         ['block6j_drop[0][0]',        
                                                                     'block6i_add[0][0]']         
                                                                                                  
 top_conv (Conv2D)           (None, 2, 2, 1408)           292864    ['block6j_add[0][0]']         
                                                                                                  
 top_bn (BatchNormalization  (None, 2, 2, 1408)           5632      ['top_conv[0][0]']            
 )                                                                                                
                                                                                                  
 top_activation (Activation  (None, 2, 2, 1408)           0         ['top_bn[0][0]']              
 )                                                                                                
                                                                                                  
==================================================================================================
Total params: 8769374 (33.45 MB)
Trainable params: 8687086 (33.14 MB)
Non-trainable params: 82288 (321.44 KB)
__________________________________________________________________________________________________
In [115]:
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"

# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)

# Prepare the data generators
train_generator = datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

validation_generator = datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,  # Keep the same batch size for simplicity
    class_mode='categorical')

test_generator = datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    shuffle=False)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Compiling and Training the Model¶

In [118]:
# Load EfficientNetB0 model pre-trained on ImageNet without the top layer
base_model = EfficientNetB0(weights='imagenet', include_top=False, input_shape=(48, 48, 3))

# Add custom layers on top of the base model
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x)
x = Dense(1024, activation='relu')(x)
x = Dropout(0.2)(x) # Adjusted intermediate layer
predictions = Dense(4, activation='softmax')(x)  # Adjust the number of units to match the number of classes

# Define the new model
efficient_model2 = Model(inputs=base_model.input, outputs=predictions)
In [119]:
efficient_model2.compile(optimizer=AdamW(learning_rate=0.000025), loss='categorical_crossentropy', metrics=['accuracy'])
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.000001)
model_checkpoint = ModelCheckpoint('best_model.h5', monitor='val_accuracy', save_best_only=True)
early_stopping = EarlyStopping(monitor='val_loss', patience=7, restore_best_weights=True)


history = efficient_model2.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // train_generator.batch_size,
    epochs=50,
    validation_data=validation_generator,
    validation_steps=validation_generator.samples // validation_generator.batch_size,
    class_weight=class_weights_adjusted,
    callbacks=[reduce_lr, model_checkpoint, early_stopping]
)
Epoch 1/50
471/472 [============================>.] - ETA: 0s - loss: 1.3364 - accuracy: 0.3444
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.
  saving_api.save_model(
472/472 [==============================] - 61s 56ms/step - loss: 1.3362 - accuracy: 0.3445 - val_loss: 1.4205 - val_accuracy: 0.2494 - lr: 2.5000e-05
Epoch 2/50
472/472 [==============================] - 24s 51ms/step - loss: 1.1789 - accuracy: 0.4744 - val_loss: 1.5377 - val_accuracy: 0.2607 - lr: 2.5000e-05
Epoch 3/50
472/472 [==============================] - 24s 50ms/step - loss: 1.0906 - accuracy: 0.5233 - val_loss: 1.3381 - val_accuracy: 0.3417 - lr: 2.5000e-05
Epoch 4/50
472/472 [==============================] - 24s 51ms/step - loss: 1.0268 - accuracy: 0.5559 - val_loss: 1.3126 - val_accuracy: 0.3831 - lr: 2.5000e-05
Epoch 5/50
472/472 [==============================] - 23s 50ms/step - loss: 0.9674 - accuracy: 0.5873 - val_loss: 1.4013 - val_accuracy: 0.3456 - lr: 2.5000e-05
Epoch 6/50
472/472 [==============================] - 24s 52ms/step - loss: 0.9249 - accuracy: 0.6089 - val_loss: 1.3047 - val_accuracy: 0.4036 - lr: 2.5000e-05
Epoch 7/50
472/472 [==============================] - 24s 51ms/step - loss: 0.8887 - accuracy: 0.6251 - val_loss: 1.2046 - val_accuracy: 0.4643 - lr: 2.5000e-05
Epoch 8/50
472/472 [==============================] - 23s 49ms/step - loss: 0.8461 - accuracy: 0.6501 - val_loss: 1.2797 - val_accuracy: 0.4194 - lr: 2.5000e-05
Epoch 9/50
472/472 [==============================] - 23s 50ms/step - loss: 0.8105 - accuracy: 0.6625 - val_loss: 1.6911 - val_accuracy: 0.2988 - lr: 2.5000e-05
Epoch 10/50
472/472 [==============================] - 23s 49ms/step - loss: 0.7762 - accuracy: 0.6786 - val_loss: 1.5159 - val_accuracy: 0.3510 - lr: 2.5000e-05
Epoch 11/50
472/472 [==============================] - 23s 49ms/step - loss: 0.7396 - accuracy: 0.6985 - val_loss: 1.3115 - val_accuracy: 0.4635 - lr: 2.5000e-05
Epoch 12/50
472/472 [==============================] - 23s 49ms/step - loss: 0.7042 - accuracy: 0.7141 - val_loss: 1.4750 - val_accuracy: 0.4284 - lr: 2.5000e-05
Epoch 13/50
472/472 [==============================] - 24s 51ms/step - loss: 0.6657 - accuracy: 0.7307 - val_loss: 1.2258 - val_accuracy: 0.4984 - lr: 5.0000e-06
Epoch 14/50
472/472 [==============================] - 24s 51ms/step - loss: 0.6549 - accuracy: 0.7325 - val_loss: 1.1271 - val_accuracy: 0.5583 - lr: 5.0000e-06
Epoch 15/50
472/472 [==============================] - 24s 51ms/step - loss: 0.6461 - accuracy: 0.7392 - val_loss: 1.0258 - val_accuracy: 0.5927 - lr: 5.0000e-06
Epoch 16/50
472/472 [==============================] - 24s 51ms/step - loss: 0.6434 - accuracy: 0.7391 - val_loss: 0.9484 - val_accuracy: 0.6177 - lr: 5.0000e-06
Epoch 17/50
472/472 [==============================] - 23s 50ms/step - loss: 0.6222 - accuracy: 0.7500 - val_loss: 1.1752 - val_accuracy: 0.5224 - lr: 5.0000e-06
Epoch 18/50
472/472 [==============================] - 23s 49ms/step - loss: 0.6238 - accuracy: 0.7521 - val_loss: 1.0146 - val_accuracy: 0.6067 - lr: 5.0000e-06
Epoch 19/50
472/472 [==============================] - 24s 50ms/step - loss: 0.6201 - accuracy: 0.7529 - val_loss: 1.0254 - val_accuracy: 0.6173 - lr: 5.0000e-06
Epoch 20/50
472/472 [==============================] - 24s 51ms/step - loss: 0.6098 - accuracy: 0.7553 - val_loss: 0.9382 - val_accuracy: 0.6367 - lr: 5.0000e-06
Epoch 21/50
472/472 [==============================] - 23s 49ms/step - loss: 0.6099 - accuracy: 0.7577 - val_loss: 1.0727 - val_accuracy: 0.5831 - lr: 5.0000e-06
Epoch 22/50
472/472 [==============================] - 24s 50ms/step - loss: 0.5948 - accuracy: 0.7632 - val_loss: 0.9776 - val_accuracy: 0.6135 - lr: 5.0000e-06
Epoch 23/50
472/472 [==============================] - 23s 49ms/step - loss: 0.5924 - accuracy: 0.7661 - val_loss: 1.0028 - val_accuracy: 0.6190 - lr: 5.0000e-06
Epoch 24/50
472/472 [==============================] - 23s 49ms/step - loss: 0.5832 - accuracy: 0.7723 - val_loss: 1.0490 - val_accuracy: 0.6113 - lr: 5.0000e-06
Epoch 25/50
472/472 [==============================] - 24s 51ms/step - loss: 0.5757 - accuracy: 0.7692 - val_loss: 0.9597 - val_accuracy: 0.6375 - lr: 5.0000e-06
Epoch 26/50
472/472 [==============================] - 24s 50ms/step - loss: 0.5759 - accuracy: 0.7736 - val_loss: 0.9805 - val_accuracy: 0.6216 - lr: 1.0000e-06
Epoch 27/50
472/472 [==============================] - 23s 50ms/step - loss: 0.5630 - accuracy: 0.7769 - val_loss: 0.9584 - val_accuracy: 0.6250 - lr: 1.0000e-06
In [120]:
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)

print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 17ms/step - loss: 0.8464 - accuracy: 0.6484
Test Loss: 0.84635990858078
Test Accuracy: 0.6484375

Evaluating the EfficientnetNet Model¶

In [123]:
plot_model_performance_and_confusion_matrix(efficient_model2, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display

# Assuming val_loss and val_accuracy are defined from your model's evaluation
# Evaluate the model on the test set
val_loss, val_accuracy = efficient_model2.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')

# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
        <p style='color: red;'><b>Test Loss:</b> efficient_model2 </p>
    <p style='color: red;'><b>Test Loss:</b> {val_loss:.2f}</p>
    <p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.2%} </p>
</div>
"""

# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.67      0.81      0.73        32
           1       0.57      0.72      0.64        32
           2       0.54      0.44      0.48        32
           3       0.87      0.62      0.73        32

    accuracy                           0.65       128
   macro avg       0.66      0.65      0.65       128
weighted avg       0.66      0.65      0.65       128

4/4 [==============================] - 0s 21ms/step - loss: 0.8464 - accuracy: 0.6484
Test Loss: 0.84635990858078
Test Accuracy: 0.6484375

Test Loss: efficient_model2

Test Loss: 0.85

Test Accuracy: 64.84%

In [ ]:
 
In [ ]:
 

Observations and Insights:__

Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance.

Think About It:

The EfficientNetB0-based model tailored for a 4-class emotion classification task exhibits an overall accuracy of 62%. Performance metrics indicate varied effectiveness across classes:

Happy class shows moderate precision and slightly higher recall, achieving a balance reflected in an F1-score of 0.70. Neutral and Sad classes have lower precision and recall, resulting in F1-scores of 0.56 and 0.56, respectively. This suggests challenges in distinguishing these emotions accurately. Surprise class demonstrates higher precision but reduced recall, with an F1-score of 0.67, indicating better identification but room for improved coverage. These results suggest that while the model can identify certain emotions with reasonable accuracy, its performance is inconsistent across all classes. The lower overall accuracy and mixed precision and recall scores highlight areas for improvement, particularly in enhancing the model's ability to recognize 'neutral' and 'sad' emotions more accurately. Optimization strategies might include adjusting the model's architecture, fine-tuning the learning rate, and employing targeted data augmentation to address these disparities.

:Observations and Insights:__

To evaluate the overall performance of the transfer learning architectures mentioned (VGG16, ResNet50, and EfficientNetB0) for the emotion classification task, let's compare their reported metrics:

VGG16-based Model:

Accuracy: 77% Best performing class: 'Surprise' with high precision and recall. Challenges: Lower precision in 'neutral' and lower recall in 'sad'. ResNet50-based Model:

Accuracy: 75% Best performing class: 'Happy' and 'Surprise' with high precision and recall. Challenges: Lower precision for 'neutral' and 'sad', indicating difficulty in distinguishing these emotions accurately. EfficientNetB0-based Model:

Accuracy: 62% Shows moderate to good performance in recognizing 'happy' and 'surprise' but struggles with 'neutral' and 'sad'. The lower overall accuracy compared to VGG16 and ResNet50.

The VGG16 and ResNet50 models have shown similar performance levels, with both achieving mid to high 70s in accuracy. They are relatively close in their ability to classify emotions, with strengths in identifying 'happy' and 'surprise' emotions accurately. The EfficientNetB0 model lagged behind in performance, with an accuracy of 62%, indicating that it might not have been as effective for this particular task or dataset. While accuracies are decent for VGG16 and ResNet50,there is still alot of room for improvement.



Conclusion:__¶

The task of computer vision is i


Pick for the best model from the two CNN's and the transfer learning models of VGG16, RESNET50 AND EFFICIENT-NET80¶

CONVOLUTIONAL NETWORK #2


In [107]:
plot_model_performance_and_confusion_matrix(cnn_model_updated, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 1s 11ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.82      0.84      0.83        32
           1       0.72      0.81      0.76        32
           2       0.79      0.69      0.73        32
           3       0.97      0.94      0.95        32

    accuracy                           0.82       128
   macro avg       0.82      0.82      0.82       128
weighted avg       0.82      0.82      0.82       128



Building a Complex Neural Network Architecture¶



Now that we have tried multiple pre-trained models, let's build a complex CNN architecture and see if we can get better performance.

In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.

Creating our Data Loaders¶

In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.

In [158]:
# Set the random Seed
np.random.seed(42)
In [159]:
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"

# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)

# Prepare the data generators
train_generator = datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

validation_generator = datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,  # Keep the same batch size for simplicity
    class_mode='categorical')

test_generator = datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    shuffle=False)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.

Specialized data augmentation to address the categorical class confusion between neutral and sad classes¶

In [160]:
from tensorflow.keras.preprocessing.image import ImageDataGenerator

# Create more augmented images for 'sad' and 'neutral' classes
augmentation_for_sad_neutral = ImageDataGenerator(
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.15,
    zoom_range=0.15,
    horizontal_flip=True,
    fill_mode='nearest'
)

Model Building¶

  • Try building a layer with 5 Convolutional Blocks and see if performance increases.
In [242]:
cnn_model_4 = Sequential([
    Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
    BatchNormalization(),
    Conv2D(64, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.3),  # Slightly reduced dropout

    # Second Convolutional Block
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.3),  # Adjusted dropout

    # Third Convolutional Block
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(512, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.4),  # Adjusted dropout

    # Fourth Convolutional Block
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.2),  # Keeping consistent dropout

     # Convolutional Block 5
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.2),  # Keeping consistent dropout


    # Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
    GlobalAveragePooling2D(),
    Flatten(),

    # Dense Layers
    Dense(128, activation='relu'),
    Dense(512, activation='relu'),
    Dropout(0.2),
    Dense(1024, activation='relu'),
    Dense(2048, activation='relu'),
    Dropout(0.2),
    Dense(128, activation='relu'),

    Dropout(0.2),
    Dense(4, activation='softmax')
])
cnn_model_4.summary()  # Print the model summary
Model: "sequential_47"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_218 (Conv2D)         (None, 48, 48, 64)        1792      
                                                                 
 batch_normalization_154 (B  (None, 48, 48, 64)        256       
 atchNormalization)                                              
                                                                 
 conv2d_219 (Conv2D)         (None, 48, 48, 64)        36928     
                                                                 
 max_pooling2d_128 (MaxPool  (None, 24, 24, 64)        0         
 ing2D)                                                          
                                                                 
 dropout_186 (Dropout)       (None, 24, 24, 64)        0         
                                                                 
 conv2d_220 (Conv2D)         (None, 24, 24, 128)       73856     
                                                                 
 batch_normalization_155 (B  (None, 24, 24, 128)       512       
 atchNormalization)                                              
                                                                 
 conv2d_221 (Conv2D)         (None, 24, 24, 256)       295168    
                                                                 
 max_pooling2d_129 (MaxPool  (None, 12, 12, 256)       0         
 ing2D)                                                          
                                                                 
 dropout_187 (Dropout)       (None, 12, 12, 256)       0         
                                                                 
 conv2d_222 (Conv2D)         (None, 12, 12, 256)       590080    
                                                                 
 batch_normalization_156 (B  (None, 12, 12, 256)       1024      
 atchNormalization)                                              
                                                                 
 conv2d_223 (Conv2D)         (None, 12, 12, 512)       1180160   
                                                                 
 batch_normalization_157 (B  (None, 12, 12, 512)       2048      
 atchNormalization)                                              
                                                                 
 max_pooling2d_130 (MaxPool  (None, 6, 6, 512)         0         
 ing2D)                                                          
                                                                 
 dropout_188 (Dropout)       (None, 6, 6, 512)         0         
                                                                 
 conv2d_224 (Conv2D)         (None, 6, 6, 1024)        4719616   
                                                                 
 batch_normalization_158 (B  (None, 6, 6, 1024)        4096      
 atchNormalization)                                              
                                                                 
 max_pooling2d_131 (MaxPool  (None, 3, 3, 1024)        0         
 ing2D)                                                          
                                                                 
 dropout_189 (Dropout)       (None, 3, 3, 1024)        0         
                                                                 
 conv2d_225 (Conv2D)         (None, 3, 3, 1024)        9438208   
                                                                 
 batch_normalization_159 (B  (None, 3, 3, 1024)        4096      
 atchNormalization)                                              
                                                                 
 max_pooling2d_132 (MaxPool  (None, 1, 1, 1024)        0         
 ing2D)                                                          
                                                                 
 dropout_190 (Dropout)       (None, 1, 1, 1024)        0         
                                                                 
 global_average_pooling2d_3  (None, 1024)              0         
 1 (GlobalAveragePooling2D)                                      
                                                                 
 flatten_34 (Flatten)        (None, 1024)              0         
                                                                 
 dense_158 (Dense)           (None, 128)               131200    
                                                                 
 dense_159 (Dense)           (None, 512)               66048     
                                                                 
 dropout_191 (Dropout)       (None, 512)               0         
                                                                 
 dense_160 (Dense)           (None, 1024)              525312    
                                                                 
 dense_161 (Dense)           (None, 2048)              2099200   
                                                                 
 dropout_192 (Dropout)       (None, 2048)              0         
                                                                 
 dense_162 (Dense)           (None, 128)               262272    
                                                                 
 dropout_193 (Dropout)       (None, 128)               0         
                                                                 
 dense_163 (Dense)           (None, 4)                 516       
                                                                 
=================================================================
Total params: 19432388 (74.13 MB)
Trainable params: 19426372 (74.11 MB)
Non-trainable params: 6016 (23.50 KB)
_________________________________________________________________
In [243]:
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)

Compiling and Training the Model¶

In [127]:
history = cnn_model_4.fit(
    train_generator,
    epochs=30,
    verbose=1,
    callbacks=[early_stopping, reduce_lr],
    validation_data=validation_generator,
    class_weight=class_weights_adjusted
)
Epoch 1/30
473/473 [==============================] - 33s 57ms/step - loss: 1.3675 - accuracy: 0.2974 - val_loss: 1.4441 - val_accuracy: 0.2465 - lr: 3.0000e-05
Epoch 2/30
473/473 [==============================] - 27s 57ms/step - loss: 1.3172 - accuracy: 0.3478 - val_loss: 1.4579 - val_accuracy: 0.2769 - lr: 3.0000e-05
Epoch 3/30
473/473 [==============================] - 28s 58ms/step - loss: 1.2367 - accuracy: 0.4090 - val_loss: 1.1947 - val_accuracy: 0.4621 - lr: 3.0000e-05
Epoch 4/30
473/473 [==============================] - 28s 59ms/step - loss: 1.1550 - accuracy: 0.4691 - val_loss: 1.0658 - val_accuracy: 0.5182 - lr: 3.0000e-05
Epoch 5/30
473/473 [==============================] - 28s 59ms/step - loss: 1.0598 - accuracy: 0.5358 - val_loss: 1.0690 - val_accuracy: 0.5339 - lr: 3.0000e-05
Epoch 6/30
473/473 [==============================] - 28s 59ms/step - loss: 0.9991 - accuracy: 0.5732 - val_loss: 0.9468 - val_accuracy: 0.6020 - lr: 3.0000e-05
Epoch 7/30
473/473 [==============================] - 28s 59ms/step - loss: 0.9407 - accuracy: 0.5997 - val_loss: 0.9827 - val_accuracy: 0.5990 - lr: 3.0000e-05
Epoch 8/30
473/473 [==============================] - 28s 59ms/step - loss: 0.8964 - accuracy: 0.6178 - val_loss: 0.9243 - val_accuracy: 0.6401 - lr: 3.0000e-05
Epoch 9/30
473/473 [==============================] - 28s 59ms/step - loss: 0.8639 - accuracy: 0.6409 - val_loss: 0.8495 - val_accuracy: 0.6703 - lr: 3.0000e-05
Epoch 10/30
473/473 [==============================] - 28s 59ms/step - loss: 0.8245 - accuracy: 0.6535 - val_loss: 0.8480 - val_accuracy: 0.6773 - lr: 3.0000e-05
Epoch 11/30
473/473 [==============================] - 28s 59ms/step - loss: 0.8008 - accuracy: 0.6653 - val_loss: 0.8084 - val_accuracy: 0.6900 - lr: 3.0000e-05
Epoch 12/30
473/473 [==============================] - 28s 59ms/step - loss: 0.7659 - accuracy: 0.6814 - val_loss: 0.7696 - val_accuracy: 0.7085 - lr: 3.0000e-05
Epoch 13/30
473/473 [==============================] - 28s 59ms/step - loss: 0.7424 - accuracy: 0.6950 - val_loss: 0.7819 - val_accuracy: 0.7012 - lr: 3.0000e-05
Epoch 14/30
473/473 [==============================] - 28s 59ms/step - loss: 0.7101 - accuracy: 0.7108 - val_loss: 0.7668 - val_accuracy: 0.6992 - lr: 3.0000e-05
Epoch 15/30
473/473 [==============================] - 28s 59ms/step - loss: 0.6990 - accuracy: 0.7163 - val_loss: 0.7166 - val_accuracy: 0.7219 - lr: 3.0000e-05
Epoch 16/30
473/473 [==============================] - 28s 59ms/step - loss: 0.6658 - accuracy: 0.7249 - val_loss: 0.6994 - val_accuracy: 0.7267 - lr: 3.0000e-05
Epoch 17/30
473/473 [==============================] - 28s 59ms/step - loss: 0.6449 - accuracy: 0.7385 - val_loss: 0.7149 - val_accuracy: 0.7312 - lr: 3.0000e-05
Epoch 18/30
473/473 [==============================] - 28s 59ms/step - loss: 0.6245 - accuracy: 0.7470 - val_loss: 0.7161 - val_accuracy: 0.7346 - lr: 3.0000e-05
Epoch 19/30
473/473 [==============================] - 28s 59ms/step - loss: 0.6038 - accuracy: 0.7531 - val_loss: 0.6973 - val_accuracy: 0.7378 - lr: 3.0000e-05
Epoch 20/30
473/473 [==============================] - 28s 59ms/step - loss: 0.5787 - accuracy: 0.7652 - val_loss: 0.7712 - val_accuracy: 0.7233 - lr: 3.0000e-05
Epoch 21/30
473/473 [==============================] - 28s 59ms/step - loss: 0.5627 - accuracy: 0.7734 - val_loss: 0.6959 - val_accuracy: 0.7380 - lr: 3.0000e-05
Epoch 22/30
473/473 [==============================] - 28s 59ms/step - loss: 0.5365 - accuracy: 0.7804 - val_loss: 0.7029 - val_accuracy: 0.7408 - lr: 3.0000e-05
Epoch 23/30
473/473 [==============================] - 28s 59ms/step - loss: 0.5206 - accuracy: 0.7927 - val_loss: 0.7007 - val_accuracy: 0.7454 - lr: 3.0000e-05
Epoch 24/30
473/473 [==============================] - 28s 59ms/step - loss: 0.4909 - accuracy: 0.8044 - val_loss: 0.7170 - val_accuracy: 0.7478 - lr: 3.0000e-05
Epoch 25/30
473/473 [==============================] - 28s 59ms/step - loss: 0.4651 - accuracy: 0.8154 - val_loss: 0.7634 - val_accuracy: 0.7370 - lr: 3.0000e-05
Epoch 26/30
473/473 [==============================] - ETA: 0s - loss: 0.4524 - accuracy: 0.8224
Epoch 26: ReduceLROnPlateau reducing learning rate to 1e-05.
473/473 [==============================] - 28s 59ms/step - loss: 0.4524 - accuracy: 0.8224 - val_loss: 0.7263 - val_accuracy: 0.7458 - lr: 3.0000e-05
Epoch 27/30
473/473 [==============================] - 28s 59ms/step - loss: 0.3850 - accuracy: 0.8508 - val_loss: 0.7633 - val_accuracy: 0.7456 - lr: 1.0000e-05
Epoch 28/30
473/473 [==============================] - 28s 59ms/step - loss: 0.3626 - accuracy: 0.8564 - val_loss: 0.7651 - val_accuracy: 0.7468 - lr: 1.0000e-05
Epoch 29/30
473/473 [==============================] - 28s 59ms/step - loss: 0.3502 - accuracy: 0.8602 - val_loss: 0.7947 - val_accuracy: 0.7454 - lr: 1.0000e-05
Epoch 30/30
473/473 [==============================] - 28s 59ms/step - loss: 0.3374 - accuracy: 0.8695 - val_loss: 0.8206 - val_accuracy: 0.7430 - lr: 1.0000e-05

TRAINING RUN 2¶

In [162]:
history = cnn_model_4.fit(
    train_generator,
    epochs=75,
    verbose=1,
    callbacks=[reduce_lr],
    validation_data=validation_generator,
    class_weight=class_weights_adjusted
    )
Epoch 1/75
473/473 [==============================] - 32s 57ms/step - loss: 1.3529 - accuracy: 0.3203 - val_loss: 1.3962 - val_accuracy: 0.3649 - lr: 3.0000e-05
Epoch 2/75
473/473 [==============================] - 27s 57ms/step - loss: 1.3064 - accuracy: 0.3630 - val_loss: 1.3467 - val_accuracy: 0.3757 - lr: 3.0000e-05
Epoch 3/75
473/473 [==============================] - 27s 58ms/step - loss: 1.2189 - accuracy: 0.4201 - val_loss: 1.1732 - val_accuracy: 0.4531 - lr: 3.0000e-05
Epoch 4/75
473/473 [==============================] - 28s 58ms/step - loss: 1.1095 - accuracy: 0.5029 - val_loss: 1.1512 - val_accuracy: 0.5142 - lr: 3.0000e-05
Epoch 5/75
473/473 [==============================] - 28s 59ms/step - loss: 1.0259 - accuracy: 0.5528 - val_loss: 1.0352 - val_accuracy: 0.5738 - lr: 3.0000e-05
Epoch 6/75
473/473 [==============================] - 28s 58ms/step - loss: 0.9608 - accuracy: 0.5862 - val_loss: 1.0371 - val_accuracy: 0.5799 - lr: 3.0000e-05
Epoch 7/75
473/473 [==============================] - 28s 58ms/step - loss: 0.9163 - accuracy: 0.6108 - val_loss: 0.9709 - val_accuracy: 0.6180 - lr: 3.0000e-05
Epoch 8/75
473/473 [==============================] - 28s 58ms/step - loss: 0.8684 - accuracy: 0.6329 - val_loss: 0.9718 - val_accuracy: 0.6339 - lr: 3.0000e-05
Epoch 9/75
473/473 [==============================] - 28s 58ms/step - loss: 0.8387 - accuracy: 0.6496 - val_loss: 0.9744 - val_accuracy: 0.6349 - lr: 3.0000e-05
Epoch 10/75
473/473 [==============================] - 28s 58ms/step - loss: 0.8003 - accuracy: 0.6652 - val_loss: 0.8822 - val_accuracy: 0.6787 - lr: 3.0000e-05
Epoch 11/75
473/473 [==============================] - 28s 59ms/step - loss: 0.7792 - accuracy: 0.6750 - val_loss: 0.8308 - val_accuracy: 0.6751 - lr: 3.0000e-05
Epoch 12/75
473/473 [==============================] - 28s 58ms/step - loss: 0.7439 - accuracy: 0.6922 - val_loss: 0.7526 - val_accuracy: 0.7028 - lr: 3.0000e-05
Epoch 13/75
473/473 [==============================] - 28s 58ms/step - loss: 0.7282 - accuracy: 0.6992 - val_loss: 0.7748 - val_accuracy: 0.6801 - lr: 3.0000e-05
Epoch 14/75
473/473 [==============================] - 28s 59ms/step - loss: 0.7001 - accuracy: 0.7132 - val_loss: 0.7531 - val_accuracy: 0.7030 - lr: 3.0000e-05
Epoch 15/75
473/473 [==============================] - 28s 58ms/step - loss: 0.6800 - accuracy: 0.7193 - val_loss: 0.7626 - val_accuracy: 0.7213 - lr: 3.0000e-05
Epoch 16/75
473/473 [==============================] - 28s 59ms/step - loss: 0.6534 - accuracy: 0.7336 - val_loss: 0.7654 - val_accuracy: 0.7189 - lr: 3.0000e-05
Epoch 17/75
473/473 [==============================] - 28s 58ms/step - loss: 0.6306 - accuracy: 0.7451 - val_loss: 0.7242 - val_accuracy: 0.7286 - lr: 3.0000e-05
Epoch 18/75
473/473 [==============================] - 28s 58ms/step - loss: 0.6145 - accuracy: 0.7500 - val_loss: 0.7546 - val_accuracy: 0.7091 - lr: 3.0000e-05
Epoch 19/75
473/473 [==============================] - 28s 58ms/step - loss: 0.5924 - accuracy: 0.7586 - val_loss: 0.6986 - val_accuracy: 0.7360 - lr: 3.0000e-05
Epoch 20/75
473/473 [==============================] - 28s 58ms/step - loss: 0.5708 - accuracy: 0.7662 - val_loss: 0.7054 - val_accuracy: 0.7362 - lr: 3.0000e-05
Epoch 21/75
473/473 [==============================] - 28s 58ms/step - loss: 0.5444 - accuracy: 0.7811 - val_loss: 0.7600 - val_accuracy: 0.7330 - lr: 3.0000e-05
Epoch 22/75
473/473 [==============================] - 28s 59ms/step - loss: 0.5204 - accuracy: 0.7886 - val_loss: 0.7341 - val_accuracy: 0.7334 - lr: 3.0000e-05
Epoch 23/75
473/473 [==============================] - 28s 58ms/step - loss: 0.4989 - accuracy: 0.7985 - val_loss: 0.7955 - val_accuracy: 0.7273 - lr: 3.0000e-05
Epoch 24/75
473/473 [==============================] - ETA: 0s - loss: 0.4749 - accuracy: 0.8068
Epoch 24: ReduceLROnPlateau reducing learning rate to 1e-05.
473/473 [==============================] - 28s 58ms/step - loss: 0.4749 - accuracy: 0.8068 - val_loss: 0.7177 - val_accuracy: 0.7454 - lr: 3.0000e-05
Epoch 25/75
473/473 [==============================] - 28s 58ms/step - loss: 0.4228 - accuracy: 0.8315 - val_loss: 0.7474 - val_accuracy: 0.7470 - lr: 1.0000e-05
Epoch 26/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3946 - accuracy: 0.8453 - val_loss: 0.7789 - val_accuracy: 0.7432 - lr: 1.0000e-05
Epoch 27/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3869 - accuracy: 0.8459 - val_loss: 0.7746 - val_accuracy: 0.7444 - lr: 1.0000e-05
Epoch 28/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3722 - accuracy: 0.8547 - val_loss: 0.7694 - val_accuracy: 0.7559 - lr: 1.0000e-05
Epoch 29/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3528 - accuracy: 0.8617 - val_loss: 0.8082 - val_accuracy: 0.7454 - lr: 1.0000e-05
Epoch 30/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3439 - accuracy: 0.8594 - val_loss: 0.7955 - val_accuracy: 0.7480 - lr: 1.0000e-05
Epoch 31/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3311 - accuracy: 0.8717 - val_loss: 0.8314 - val_accuracy: 0.7464 - lr: 1.0000e-05
Epoch 32/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3227 - accuracy: 0.8716 - val_loss: 0.8423 - val_accuracy: 0.7406 - lr: 1.0000e-05
Epoch 33/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3145 - accuracy: 0.8768 - val_loss: 0.8531 - val_accuracy: 0.7446 - lr: 1.0000e-05
Epoch 34/75
473/473 [==============================] - 28s 58ms/step - loss: 0.3001 - accuracy: 0.8834 - val_loss: 0.8843 - val_accuracy: 0.7482 - lr: 1.0000e-05
Epoch 35/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2829 - accuracy: 0.8889 - val_loss: 0.9057 - val_accuracy: 0.7402 - lr: 1.0000e-05
Epoch 36/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2840 - accuracy: 0.8887 - val_loss: 0.8644 - val_accuracy: 0.7440 - lr: 1.0000e-05
Epoch 37/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2776 - accuracy: 0.8927 - val_loss: 0.8776 - val_accuracy: 0.7388 - lr: 1.0000e-05
Epoch 38/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2650 - accuracy: 0.8952 - val_loss: 0.9019 - val_accuracy: 0.7422 - lr: 1.0000e-05
Epoch 39/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2612 - accuracy: 0.8976 - val_loss: 0.9259 - val_accuracy: 0.7448 - lr: 1.0000e-05
Epoch 40/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2482 - accuracy: 0.9060 - val_loss: 0.9300 - val_accuracy: 0.7476 - lr: 1.0000e-05
Epoch 41/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2518 - accuracy: 0.9027 - val_loss: 0.8982 - val_accuracy: 0.7476 - lr: 1.0000e-05
Epoch 42/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2349 - accuracy: 0.9086 - val_loss: 0.9488 - val_accuracy: 0.7470 - lr: 1.0000e-05
Epoch 43/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2288 - accuracy: 0.9130 - val_loss: 0.9425 - val_accuracy: 0.7519 - lr: 1.0000e-05
Epoch 44/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2199 - accuracy: 0.9153 - val_loss: 0.9910 - val_accuracy: 0.7416 - lr: 1.0000e-05
Epoch 45/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2143 - accuracy: 0.9180 - val_loss: 0.9927 - val_accuracy: 0.7480 - lr: 1.0000e-05
Epoch 46/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2127 - accuracy: 0.9192 - val_loss: 0.9895 - val_accuracy: 0.7498 - lr: 1.0000e-05
Epoch 47/75
473/473 [==============================] - 28s 58ms/step - loss: 0.2106 - accuracy: 0.9196 - val_loss: 1.0095 - val_accuracy: 0.7438 - lr: 1.0000e-05
Epoch 48/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1981 - accuracy: 0.9259 - val_loss: 1.0120 - val_accuracy: 0.7505 - lr: 1.0000e-05
Epoch 49/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1980 - accuracy: 0.9231 - val_loss: 0.9901 - val_accuracy: 0.7490 - lr: 1.0000e-05
Epoch 50/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1898 - accuracy: 0.9265 - val_loss: 1.0584 - val_accuracy: 0.7418 - lr: 1.0000e-05
Epoch 51/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1845 - accuracy: 0.9279 - val_loss: 1.0573 - val_accuracy: 0.7533 - lr: 1.0000e-05
Epoch 52/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1808 - accuracy: 0.9321 - val_loss: 1.0527 - val_accuracy: 0.7472 - lr: 1.0000e-05
Epoch 53/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1741 - accuracy: 0.9338 - val_loss: 1.0805 - val_accuracy: 0.7430 - lr: 1.0000e-05
Epoch 54/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1675 - accuracy: 0.9385 - val_loss: 1.0889 - val_accuracy: 0.7458 - lr: 1.0000e-05
Epoch 55/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1632 - accuracy: 0.9375 - val_loss: 1.0601 - val_accuracy: 0.7454 - lr: 1.0000e-05
Epoch 56/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1649 - accuracy: 0.9381 - val_loss: 1.0953 - val_accuracy: 0.7438 - lr: 1.0000e-05
Epoch 57/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1606 - accuracy: 0.9389 - val_loss: 1.0946 - val_accuracy: 0.7482 - lr: 1.0000e-05
Epoch 58/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1558 - accuracy: 0.9386 - val_loss: 1.1140 - val_accuracy: 0.7452 - lr: 1.0000e-05
Epoch 59/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1515 - accuracy: 0.9437 - val_loss: 1.1096 - val_accuracy: 0.7472 - lr: 1.0000e-05
Epoch 60/75
473/473 [==============================] - 28s 59ms/step - loss: 0.1411 - accuracy: 0.9466 - val_loss: 1.1481 - val_accuracy: 0.7390 - lr: 1.0000e-05
Epoch 61/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1419 - accuracy: 0.9473 - val_loss: 1.1623 - val_accuracy: 0.7410 - lr: 1.0000e-05
Epoch 62/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1389 - accuracy: 0.9478 - val_loss: 1.1749 - val_accuracy: 0.7426 - lr: 1.0000e-05
Epoch 63/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1333 - accuracy: 0.9508 - val_loss: 1.1665 - val_accuracy: 0.7472 - lr: 1.0000e-05
Epoch 64/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1287 - accuracy: 0.9514 - val_loss: 1.1912 - val_accuracy: 0.7503 - lr: 1.0000e-05
Epoch 65/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1321 - accuracy: 0.9507 - val_loss: 1.1520 - val_accuracy: 0.7386 - lr: 1.0000e-05
Epoch 66/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1255 - accuracy: 0.9532 - val_loss: 1.1690 - val_accuracy: 0.7535 - lr: 1.0000e-05
Epoch 67/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1196 - accuracy: 0.9538 - val_loss: 1.1468 - val_accuracy: 0.7527 - lr: 1.0000e-05
Epoch 68/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1157 - accuracy: 0.9570 - val_loss: 1.2080 - val_accuracy: 0.7444 - lr: 1.0000e-05
Epoch 69/75
473/473 [==============================] - 28s 59ms/step - loss: 0.1148 - accuracy: 0.9577 - val_loss: 1.2205 - val_accuracy: 0.7420 - lr: 1.0000e-05
Epoch 70/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1019 - accuracy: 0.9632 - val_loss: 1.2842 - val_accuracy: 0.7430 - lr: 1.0000e-05
Epoch 71/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1089 - accuracy: 0.9602 - val_loss: 1.2643 - val_accuracy: 0.7450 - lr: 1.0000e-05
Epoch 72/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1109 - accuracy: 0.9591 - val_loss: 1.2686 - val_accuracy: 0.7517 - lr: 1.0000e-05
Epoch 73/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1053 - accuracy: 0.9628 - val_loss: 1.2550 - val_accuracy: 0.7428 - lr: 1.0000e-05
Epoch 74/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1052 - accuracy: 0.9600 - val_loss: 1.2497 - val_accuracy: 0.7456 - lr: 1.0000e-05
Epoch 75/75
473/473 [==============================] - 28s 58ms/step - loss: 0.1007 - accuracy: 0.9621 - val_loss: 1.2674 - val_accuracy: 0.7392 - lr: 1.0000e-05

Evaluating the Model on Test Set¶

In [157]:
plot_model_performance_and_confusion_matrix(cnn_model_4, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
from IPython.display import HTML, display

val_loss, val_accuracy = cnn_model_4.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)
print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')

# Format the evaluation results as HTML
output_html = f"""
<div style='background-color: #f0f0f5; text-align: center; font-size: 30px; padding: 20px; border-radius: 10px;'>
        <p style='color: red;'><b>Test Loss:</b> CNN_MODEL_4 </p>
    <p style='color: red;'><b>Test Loss:</b> {val_loss:.4f}</p>
    <p style='color: blue;'><b>Test Accuracy:</b> {val_accuracy:.4%} </p>
</div>
"""

# Display the styled output
display(HTML(output_html))
4/4 [==============================] - 0s 17ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.86      0.78      0.82        32
           1       0.65      0.75      0.70        32
           2       0.70      0.66      0.68        32
           3       0.84      0.84      0.84        32

    accuracy                           0.76       128
   macro avg       0.76      0.76      0.76       128
weighted avg       0.76      0.76      0.76       128

4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578
Test Loss: 0.6132991909980774
Test Accuracy: 0.7578125

Test Loss: CNN_MODEL_4

Test Loss: 0.6133

Test Accuracy: 75.7812%

Plotting the Confusion Matrix for the DEEP-CONVOLUTIONAL-NEURAL-NETWORK-CNN_MODEL¶


In [129]:
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)
# correcting mislabeling
true_labels = test_generator.classes

# Get the class names
class_names_list = test_generator.class_indices.keys()
print(class_names_list)

# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)

# Plot the confusion matrix
cm = confusion_matrix(true_labels, predicted_labels)
4/4 [==============================] - 0s 17ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
              precision    recall  f1-score   support

           0       0.96      0.81      0.88        32
           1       0.60      0.81      0.69        32
           2       0.71      0.69      0.70        32
           3       1.00      0.84      0.92        32

    accuracy                           0.79       128
   macro avg       0.82      0.79      0.80       128
weighted avg       0.82      0.79      0.80       128

Insights¶

Refined insights:¶

  • What are the most meaningful insights from the data relevant to the problem?

Comparison of various techniques and their relative performance:¶

  • How do different techniques perform? Which one is performing relatively better? Is there scope to improve the performance further?

Proposal for the final solution design:¶

  • What model do you propose to be adopted? Why is this the best solution to adopt?

CNN_MODEL_5 - 'SIAMESE' NETWORK-¶

  • To over come the dispairity in the data set, i have built a model that is trained to distinguish between similar and dissimilar image parts.
  • It is then used to perform feature extraction on the images in the pipeline

  • the Siamese network outputs the features from the last layer before the final similarity score calculation.

  • Then those output feature vectors are used as the inputs to the CNN_MODEL_5

  • Start with defining the input shape for our images, making sure it matches the resolution we're working with, like (48, 48, 3) for 48x48 RGB images.

  • Create the base CNN model that learns feature representations from the images. This model is crucial as it captures the essence of the images.

  • Define two input tensors for the Siamese network, representing the pair of images to compare.

  • Reuse the base CNN model for both inputs to ensure the feature extraction process is identical for each image in the pair.

  • Process both inputs through the base CNN independently, obtaining their feature representations.

  • Calculate the distance between the two sets of features to understand how similar or dissimilar the images are. The absolute difference works well for this.

  • Feed the distance through a dense layer with a sigmoid activation to get a similarity score, indicating the likelihood of the images being the same class. Compile the Siamese network, choosing an optimizer and a loss function tailored for comparison tasks, like contrastive loss.

  • Prepare the image pairs for training, ensuring we have both positive pairs (same class) and negative pairs (different classes).

  • Augment the dataset if needed, especially to address class imbalance or to increase the diversity of training examples.

  • Train the Siamese network on these pairs, focusing on learning the subtle differences and similarities between classes.

  • Evaluate the model on a separate validation set to ensure it generalizes well to new, unseen image pairs.

This Model is trained to differentiate between similar and dissimilar image pairs, perfect for tasks requiring similarity or dissimilarity assessments. In contrast, my CNN model (cnn_model_4) is tailored for direct classification tasks across multiple categories, specifically four output classes in my case.¶

  • Effective Leveraging of Both Models:
  • Feature Extraction and Comparison:
  • Siamese for Feature Vectors: I use the trained Siamese model, or specifically its base CNN part, to extract feature vectors from images. This involves modifying the Siamese network to output features from the layer just before the final similarity score calculation.

  • CNN for Classification: The extracted feature vectors serve as inputs to cnn_model_4 or a similar classification model. This requires me to adjust cnn_model_4 to accept the feature vector size as its input shape, possibly changing the first layer to a Dense layer from a Conv2D, or using the features directly in a custom training loop to influence the classification decision. Using Siamese Model for Data Augmentation:

  • Balance the Dataset: I employ the Siamese model to create a more balanced training dataset by identifying similar pairs within underrepresented classes and dissimilar pairs across classes, thereby augmenting the dataset to balance class representation.
  • Enhance CNN Training: cnn_model_4 is then trained on this augmented, more balanced dataset to improve its generalization ability and performance on minority classes.
  • Transfer Learning:
  • Feature Transfer: I utilize the base CNN from the Siamese network as a pre-trained feature extractor for cnn_model_4. This involves replacing the initial layers of cnn_model_4 with the base CNN layers from the Siamese model, and possibly freezing some or all of these layers during initial training phases.
  • Fine-Tuning: After the initial training phase with some layers frozen, I consider fine-tuning the entire model (cnn_model_4 with the Siamese base CNN layers) by training it end-to-end with a very low learning rate to better adapt the pre-trained features to my specific classification task. Practical Steps:
  • Extract Base CNN from Siamese Model: If not already done, I extract the base CNN model used in the Siamese network. This model is crucial as it learns rich feature representations beneficial for both similarity assessment and classification.
  • Integrate Base CNN into cnn_model_4:

Base CNN Model¶

In [223]:
from keras.preprocessing.image import ImageDataGenerator

train_datagen = ImageDataGenerator(
    rescale=1./255,
    rotation_range=20,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=False,
    fill_mode='nearest'
)
# Define the image size and paths
image_size = 48
train_path = "Facial_emotion_images/train"
test_path = "Facial_emotion_images/test"
validation_path = "Facial_emotion_images/validation"

# Initialize the data generator with rescaling
datagen = ImageDataGenerator(rescale=1./255)

# Prepare the data generators
train_generator = datagen.flow_from_directory(
    train_path,
    target_size=(48, 48),
    batch_size=32,
    class_mode='categorical')

validation_generator = datagen.flow_from_directory(
    validation_path,
    target_size=(48, 48),
    batch_size=32,  # Keep the same batch size for simplicity
    class_mode='categorical')

test_generator = datagen.flow_from_directory(
    test_path,
    target_size=(48, 48),
    shuffle=False)
Found 15109 images belonging to 4 classes.
Found 4977 images belonging to 4 classes.
Found 128 images belonging to 4 classes.
In [233]:
cnn_model_4 = Sequential([
    Conv2D(64, (3, 3), activation='relu', padding='same', input_shape=(48, 48, 3)),
    BatchNormalization(),
    Conv2D(64, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.3),  # Slightly reduced dropout

    # Second Convolutional Block
    Conv2D(128, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    MaxPooling2D((2, 2)),
    Dropout(0.3),  # Adjusted dropout

    # Third Convolutional Block
    Conv2D(256, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    Conv2D(512, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.4),  # Adjusted dropout

    # Fourth Convolutional Block
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.2),  # Keeping consistent dropout

     # Convolutional Block 5
    Conv2D(1024, (3, 3), activation='relu', padding='same'),
    BatchNormalization(),
    MaxPooling2D((2, 2)),
    Dropout(0.2),  # Keeping consistent dropout


    # Global Average Pooling replaces Flatten to reduce parameters and potential overfitting
    GlobalAveragePooling2D(),
    Flatten(),

    # Dense Layers
    Dense(128, activation='relu'),
    Dense(512, activation='relu'),
    Dropout(0.2),
    Dense(1024, activation='relu'),
    Dense(2048, activation='relu'),
    Dropout(0.4),
    Dense(128, activation='relu'),

    Dropout(0.2),
    Dense(4, activation='softmax')
])
cnn_model_4.compile(optimizer=AdamW(learning_rate=0.00003),
              loss='categorical_crossentropy',
              metrics=['accuracy'])

early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)  # Increased patience
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)  # Adjusted patience
cnn_model_4.summary()
Model: "sequential_43"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_210 (Conv2D)         (None, 48, 48, 64)        1792      
                                                                 
 batch_normalization_148 (B  (None, 48, 48, 64)        256       
 atchNormalization)                                              
                                                                 
 conv2d_211 (Conv2D)         (None, 48, 48, 64)        36928     
                                                                 
 max_pooling2d_123 (MaxPool  (None, 24, 24, 64)        0         
 ing2D)                                                          
                                                                 
 dropout_178 (Dropout)       (None, 24, 24, 64)        0         
                                                                 
 conv2d_212 (Conv2D)         (None, 24, 24, 128)       73856     
                                                                 
 batch_normalization_149 (B  (None, 24, 24, 128)       512       
 atchNormalization)                                              
                                                                 
 conv2d_213 (Conv2D)         (None, 24, 24, 256)       295168    
                                                                 
 max_pooling2d_124 (MaxPool  (None, 12, 12, 256)       0         
 ing2D)                                                          
                                                                 
 dropout_179 (Dropout)       (None, 12, 12, 256)       0         
                                                                 
 conv2d_214 (Conv2D)         (None, 12, 12, 256)       590080    
                                                                 
 batch_normalization_150 (B  (None, 12, 12, 256)       1024      
 atchNormalization)                                              
                                                                 
 conv2d_215 (Conv2D)         (None, 12, 12, 512)       1180160   
                                                                 
 batch_normalization_151 (B  (None, 12, 12, 512)       2048      
 atchNormalization)                                              
                                                                 
 max_pooling2d_125 (MaxPool  (None, 6, 6, 512)         0         
 ing2D)                                                          
                                                                 
 dropout_180 (Dropout)       (None, 6, 6, 512)         0         
                                                                 
 conv2d_216 (Conv2D)         (None, 6, 6, 1024)        4719616   
                                                                 
 batch_normalization_152 (B  (None, 6, 6, 1024)        4096      
 atchNormalization)                                              
                                                                 
 max_pooling2d_126 (MaxPool  (None, 3, 3, 1024)        0         
 ing2D)                                                          
                                                                 
 dropout_181 (Dropout)       (None, 3, 3, 1024)        0         
                                                                 
 conv2d_217 (Conv2D)         (None, 3, 3, 1024)        9438208   
                                                                 
 batch_normalization_153 (B  (None, 3, 3, 1024)        4096      
 atchNormalization)                                              
                                                                 
 max_pooling2d_127 (MaxPool  (None, 1, 1, 1024)        0         
 ing2D)                                                          
                                                                 
 dropout_182 (Dropout)       (None, 1, 1, 1024)        0         
                                                                 
 global_average_pooling2d_3  (None, 1024)              0         
 0 (GlobalAveragePooling2D)                                      
                                                                 
 flatten_30 (Flatten)        (None, 1024)              0         
                                                                 
 dense_146 (Dense)           (None, 128)               131200    
                                                                 
 dense_147 (Dense)           (None, 512)               66048     
                                                                 
 dropout_183 (Dropout)       (None, 512)               0         
                                                                 
 dense_148 (Dense)           (None, 1024)              525312    
                                                                 
 dense_149 (Dense)           (None, 2048)              2099200   
                                                                 
 dropout_184 (Dropout)       (None, 2048)              0         
                                                                 
 dense_150 (Dense)           (None, 128)               262272    
                                                                 
 dropout_185 (Dropout)       (None, 128)               0         
                                                                 
 dense_151 (Dense)           (None, 4)                 516       
                                                                 
=================================================================
Total params: 19432388 (74.13 MB)
Trainable params: 19426372 (74.11 MB)
Non-trainable params: 6016 (23.50 KB)
_________________________________________________________________

Adjust based on the feature_model's output shape¶

In [255]:
input_shape = (48, 48, 3)
base_cnn = create_base_cnn(input_shape)
feature_model = Model(inputs=base_cnn.input, outputs=base_cnn.layers[-5].output)
In [256]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dropout, GlobalAveragePooling2D, Dense, BatchNormalization, Flatten
from tensorflow.keras.regularizers import l2

def create_base_cnn(input_shape):
    model = Sequential([
        Conv2D(64, (3, 3), activation='relu', input_shape=input_shape, padding='same'),
        BatchNormalization(),
        Conv2D(64, (3, 3), activation='relu', padding='same'),
        MaxPooling2D((2, 2)),
        Dropout(0.3),

        Conv2D(128, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        Conv2D(256, (3, 3), activation='relu', padding='same'),
        MaxPooling2D((2, 2)),
        Dropout(0.3),

        Conv2D(256, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        Conv2D(512, (3, 3), activation='relu', padding='same'),
        BatchNormalization(),
        MaxPooling2D((2, 2)),
        Dropout(0.4),

        GlobalAveragePooling2D(),

        Dense(512, activation='relu'),
        Dropout(0.4),
        Dense(512, activation='relu', kernel_regularizer=l2(0.001)),
    ])
    return model
In [257]:
from tensorflow.keras.layers import Input, Lambda
from tensorflow.keras.models import Model
import tensorflow.keras.backend as K
input_shape = (48, 48, 3)
def create_siamese_network(input_shape):
    input_a = Input(shape=input_shape)
    input_b = Input(shape=input_shape)

    base_cnn = create_base_cnn(input_shape)

    processed_a = base_cnn(input_a)
    processed_b = base_cnn(input_b)

    distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
    outputs = Dense(1, activation='sigmoid')(distance)
    model = Model(inputs=[input_a, input_b], outputs=outputs)

    return model

Set the Callbacks¶

In [260]:
import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Conv2D, MaxPooling2D, Flatten, Dense, Lambda
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
from sklearn.model_selection import train_test_split

Fitting the Model¶

In [266]:
def create_base_cnn(input_shape):
    input = Input(shape=input_shape)
    x = Conv2D(64, (3, 3), activation='relu')(input)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Conv2D(128, (3, 3), activation='relu')(x)
    x = MaxPooling2D(pool_size=(2, 2))(x)
    x = Flatten()(x)
    x = Dense(128, activation='relu')(x)
    return Model(inputs=input, outputs=x)
In [267]:
def create_siamese_network(input_shape):
    base_network = create_base_cnn(input_shape)

    input_a = Input(shape=input_shape)
    input_b = Input(shape=input_shape)

    processed_a = base_network(input_a)
    processed_b = base_network(input_b)

    distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]))([processed_a, processed_b])
    outputs = Dense(1, activation='sigmoid')(distance)
    model = Model(inputs=[input_a, input_b], outputs=outputs)

    return model
In [277]:
input_shape = (48, 48, 3)  # Adjust based on your dataset
siamese_network = create_siamese_network(input_shape)
siamese_network.compile(optimizer=Adam(0.0001), loss='binary_crossentropy', metrics=['accuracy'])
In [ ]:
# Define callbacks
early_stopping = EarlyStopping(monitor='val_loss', patience=10, verbose=1, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, verbose=1, min_lr=0.00001)

# Fit the model
history = cnn_model_4.fit(
    train_generator,
    steps_per_epoch=train_generator.samples // train_generator.batch_size,
    epochs=100,
    validation_data=validation_generator,# Adjusted data loading
    validation_steps=validation_generator.samples // validation_generator.batch_size,# adjusted data loading
    callbacks=[early_stopping, reduce_lr],
    verbose=1
)
Epoch 1/100
472/472 [==============================] - 25s 43ms/step - loss: 1.3865 - accuracy: 0.3460 - val_loss: 1.6837 - val_accuracy: 0.3663 - lr: 3.0000e-05
Epoch 2/100
472/472 [==============================] - 20s 43ms/step - loss: 1.2545 - accuracy: 0.4210 - val_loss: 1.5015 - val_accuracy: 0.3835 - lr: 3.0000e-05
Epoch 3/100
472/472 [==============================] - 20s 43ms/step - loss: 1.1560 - accuracy: 0.4819 - val_loss: 1.3127 - val_accuracy: 0.4194 - lr: 3.0000e-05
Epoch 4/100
472/472 [==============================] - 20s 43ms/step - loss: 1.0680 - accuracy: 0.5405 - val_loss: 1.1432 - val_accuracy: 0.5143 - lr: 3.0000e-05
Epoch 5/100
472/472 [==============================] - 20s 42ms/step - loss: 1.0107 - accuracy: 0.5706 - val_loss: 0.9674 - val_accuracy: 0.5817 - lr: 3.0000e-05
Epoch 6/100
472/472 [==============================] - 20s 42ms/step - loss: 0.9745 - accuracy: 0.5841 - val_loss: 1.1597 - val_accuracy: 0.5379 - lr: 3.0000e-05
Epoch 7/100
472/472 [==============================] - 20s 43ms/step - loss: 0.9306 - accuracy: 0.6095 - val_loss: 0.9213 - val_accuracy: 0.6222 - lr: 3.0000e-05
Epoch 8/100
472/472 [==============================] - 20s 43ms/step - loss: 0.8935 - accuracy: 0.6266 - val_loss: 0.8629 - val_accuracy: 0.6468 - lr: 3.0000e-05
Epoch 9/100
472/472 [==============================] - 20s 43ms/step - loss: 0.8591 - accuracy: 0.6397 - val_loss: 0.9322 - val_accuracy: 0.6288 - lr: 3.0000e-05
Epoch 10/100
472/472 [==============================] - 20s 42ms/step - loss: 0.8364 - accuracy: 0.6540 - val_loss: 0.8387 - val_accuracy: 0.6633 - lr: 3.0000e-05
Epoch 11/100
472/472 [==============================] - 20s 43ms/step - loss: 0.8130 - accuracy: 0.6615 - val_loss: 0.8049 - val_accuracy: 0.6722 - lr: 3.0000e-05
Epoch 12/100
472/472 [==============================] - 20s 42ms/step - loss: 0.7941 - accuracy: 0.6763 - val_loss: 0.7672 - val_accuracy: 0.7006 - lr: 3.0000e-05
Epoch 13/100
472/472 [==============================] - 20s 43ms/step - loss: 0.7713 - accuracy: 0.6821 - val_loss: 0.7706 - val_accuracy: 0.6976 - lr: 3.0000e-05
Epoch 14/100
472/472 [==============================] - 20s 43ms/step - loss: 0.7454 - accuracy: 0.6914 - val_loss: 0.7470 - val_accuracy: 0.7002 - lr: 3.0000e-05
Epoch 15/100
472/472 [==============================] - 20s 43ms/step - loss: 0.7268 - accuracy: 0.7003 - val_loss: 0.7572 - val_accuracy: 0.6978 - lr: 3.0000e-05
Epoch 16/100
472/472 [==============================] - 20s 43ms/step - loss: 0.7109 - accuracy: 0.7088 - val_loss: 0.7452 - val_accuracy: 0.7026 - lr: 3.0000e-05
Epoch 17/100
472/472 [==============================] - 20s 43ms/step - loss: 0.6928 - accuracy: 0.7184 - val_loss: 0.7401 - val_accuracy: 0.7079 - lr: 3.0000e-05
Epoch 18/100
472/472 [==============================] - 20s 43ms/step - loss: 0.6757 - accuracy: 0.7249 - val_loss: 0.7120 - val_accuracy: 0.7216 - lr: 3.0000e-05
Epoch 19/100
472/472 [==============================] - 21s 44ms/step - loss: 0.6559 - accuracy: 0.7340 - val_loss: 0.6963 - val_accuracy: 0.7278 - lr: 3.0000e-05
Epoch 20/100
472/472 [==============================] - 20s 43ms/step - loss: 0.6376 - accuracy: 0.7405 - val_loss: 0.6986 - val_accuracy: 0.7262 - lr: 3.0000e-05
Epoch 21/100
472/472 [==============================] - 20s 43ms/step - loss: 0.6212 - accuracy: 0.7464 - val_loss: 0.7125 - val_accuracy: 0.7254 - lr: 3.0000e-05
Epoch 22/100
472/472 [==============================] - 20s 43ms/step - loss: 0.6017 - accuracy: 0.7613 - val_loss: 0.6822 - val_accuracy: 0.7323 - lr: 3.0000e-05
Epoch 23/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5885 - accuracy: 0.7675 - val_loss: 0.6721 - val_accuracy: 0.7397 - lr: 3.0000e-05
Epoch 24/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5641 - accuracy: 0.7730 - val_loss: 0.6693 - val_accuracy: 0.7454 - lr: 3.0000e-05
Epoch 25/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5524 - accuracy: 0.7795 - val_loss: 0.6849 - val_accuracy: 0.7361 - lr: 3.0000e-05
Epoch 26/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5410 - accuracy: 0.7822 - val_loss: 0.6808 - val_accuracy: 0.7345 - lr: 3.0000e-05
Epoch 27/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5218 - accuracy: 0.7951 - val_loss: 0.6800 - val_accuracy: 0.7399 - lr: 3.0000e-05
Epoch 28/100
472/472 [==============================] - 20s 43ms/step - loss: 0.5034 - accuracy: 0.8005 - val_loss: 0.6799 - val_accuracy: 0.7407 - lr: 3.0000e-05
Epoch 29/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4944 - accuracy: 0.8042 - val_loss: 0.6570 - val_accuracy: 0.7532 - lr: 3.0000e-05
Epoch 30/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4700 - accuracy: 0.8151 - val_loss: 0.6590 - val_accuracy: 0.7516 - lr: 3.0000e-05
Epoch 31/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4588 - accuracy: 0.8213 - val_loss: 0.6479 - val_accuracy: 0.7506 - lr: 3.0000e-05
Epoch 32/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4400 - accuracy: 0.8301 - val_loss: 0.6796 - val_accuracy: 0.7468 - lr: 3.0000e-05
Epoch 33/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4303 - accuracy: 0.8328 - val_loss: 0.6759 - val_accuracy: 0.7433 - lr: 3.0000e-05
Epoch 34/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4136 - accuracy: 0.8395 - val_loss: 0.6724 - val_accuracy: 0.7496 - lr: 3.0000e-05
Epoch 35/100
472/472 [==============================] - 20s 43ms/step - loss: 0.4050 - accuracy: 0.8461 - val_loss: 0.7359 - val_accuracy: 0.7252 - lr: 3.0000e-05
Epoch 36/100
471/472 [============================>.] - ETA: 0s - loss: 0.3864 - accuracy: 0.8512
Epoch 36: ReduceLROnPlateau reducing learning rate to 1e-05.
472/472 [==============================] - 20s 43ms/step - loss: 0.3865 - accuracy: 0.8512 - val_loss: 0.6757 - val_accuracy: 0.7506 - lr: 3.0000e-05
Epoch 37/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3486 - accuracy: 0.8706 - val_loss: 0.6466 - val_accuracy: 0.7641 - lr: 1.0000e-05
Epoch 38/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3408 - accuracy: 0.8740 - val_loss: 0.6653 - val_accuracy: 0.7571 - lr: 1.0000e-05
Epoch 39/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3252 - accuracy: 0.8841 - val_loss: 0.6793 - val_accuracy: 0.7526 - lr: 1.0000e-05
Epoch 40/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3270 - accuracy: 0.8804 - val_loss: 0.6592 - val_accuracy: 0.7647 - lr: 1.0000e-05
Epoch 41/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3145 - accuracy: 0.8842 - val_loss: 0.6528 - val_accuracy: 0.7663 - lr: 1.0000e-05
Epoch 42/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3107 - accuracy: 0.8898 - val_loss: 0.6633 - val_accuracy: 0.7649 - lr: 1.0000e-05
Epoch 43/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3040 - accuracy: 0.8900 - val_loss: 0.6761 - val_accuracy: 0.7603 - lr: 1.0000e-05
Epoch 44/100
472/472 [==============================] - 20s 43ms/step - loss: 0.3054 - accuracy: 0.8869 - val_loss: 0.6615 - val_accuracy: 0.7637 - lr: 1.0000e-05
Epoch 45/100
472/472 [==============================] - 20s 43ms/step - loss: 0.2913 - accuracy: 0.8928 - val_loss: 0.6561 - val_accuracy: 0.7681 - lr: 1.0000e-05
Epoch 46/100
472/472 [==============================] - 20s 43ms/step - loss: 0.2858 - accuracy: 0.8963 - val_loss: 0.6878 - val_accuracy: 0.7579 - lr: 1.0000e-05
Epoch 47/100
471/472 [============================>.] - ETA: 0s - loss: 0.2852 - accuracy: 0.8956Restoring model weights from the end of the best epoch: 37.
472/472 [==============================] - 20s 43ms/step - loss: 0.2857 - accuracy: 0.8954 - val_loss: 0.6985 - val_accuracy: 0.7550 - lr: 1.0000e-05
Epoch 47: early stopping

Evaluate the model on the test set¶

In [ ]:
# Evaluate the model on the test set
val_loss, val_accuracy = cnn_model_4.evaluate(
    test_generator,
    steps=test_generator.samples // test_generator.batch_size
)

print(f'Test Loss: {val_loss}')
print(f'Test Accuracy: {val_accuracy}')
4/4 [==============================] - 0s 16ms/step - loss: 0.5081 - accuracy: 0.7969
Test Loss: 0.5081095695495605
Test Accuracy: 0.796875
In [ ]:
from sklearn.metrics import confusion_matrix, classification_report

# Plot the training and validation accuracy values
plt.figure(figsize=(35, 6))
plt.subplot(1, 4, 1)
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('cnn_model_4 Accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 4, 2)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('cnn_model_4 Loss')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Validation'], loc='upper left')

# Plot the training and validation loss values
plt.subplot(1, 4, 3)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues", cbar=False)
plt.xlabel("Predicted Label")
plt.ylabel("True Label")
plt.title("Confusion Matrix for cnn_model_4")


predicted_labels = np.argmax(cnn_model_4.predict(test_generator), axis=1)

# Get the true labels for the test set
true_labels = test_generator.classes

# Get the class names
class_names_list = test_generator.class_indices.keys()

# Calculate and display the metrics
metrics_score(true_labels, predicted_labels)
plt.show()
4/4 [==============================] - 0s 16ms/step
              precision    recall  f1-score   support

           0       0.83      0.91      0.87        32
           1       0.69      0.75      0.72        32
           2       0.72      0.66      0.69        32
           3       0.97      0.88      0.92        32

    accuracy                           0.80       128
   macro avg       0.80      0.80      0.80       128
weighted avg       0.80      0.80      0.80       128

In [ ]:
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

The last model did not reach the performance levels that were expected; however, with some time, hyperparameter tuning, and adjustments to the model parameters, I believe that progress can be made towards achieving higher performance by reducing the confusion between the categorical classes of sad and neutral. The primary conclusion of this project is that for the task of creating a facial recognition and emotion classification model, a 5-layer convolutional neural network with sufficiently deep dense layers and a proper convolution setup can be used for a range of applications depending on the use case. However, the performance levels reached in this notebook top out at 82%. Production-grade applications will require much higher performance than was accomplished here.¶

However, the methods used show that substantial gains can be made:

Data augmentation for the training data helps to make the smaller dataset more robust. Weighting the classes helped with making the class distribution skew less detrimental to the overall performance. Transfer learning on the dataset proved to be difficult; however, after a good amount of hyperparameter adjusting, two of the three transfer learning models started to show promising results. The second convolutional neural network constructed is the overall best performer among the models built here.

The final model tries to extract useful feature vector data and then fuse its learned properties with the training process of a standard CNN. This is an attempt to take a more specialized method to really learn and embed the deep underlying properties that differentiate the classes of neutral and sad. Those two classes gave all the models the most difficulty during testing, as seen in the included confusion matrices for each model's testing results.

This task is important. Being able to detect basic emotions rests upon the base technologies that will enable more impactful use cases, such as driver monitoring in automobiles, to check if the driver is paying attention. This is especially important with level 2 and 3 semi-autonomous vehicles where there is a notable vigilance decrement by drivers. Drivers can get the sense that the car is fully in control and will start not only to not pay attention but, in some extreme cases, even fall asleep. The methods to be able to detect a human face and determine the emotion of an individual can easily be extended and used to track the driver's attention span and ensure that they are ready to take back control of the vehicle.

To conclude-- Executive Summary¶

In this project, we developed a facial recognition and emotion classification model using a convolutional neural network (CNN). The model, a 5-layer CNN with deep dense layers and optimized convolution setup, achieved 82% accuracy.

The main challenge was accurately classifying facial emotions due to the subtle differences between expressions. This is critical for sectors like security and customer service, where understanding human emotions is key.

The chosen CNN architecture relys on deep learning's pattern recognition capabilities, especially for image data, enhancing the model's ability to detect subtle emotional features. Implementing this model could significantly improve human-computer interactions, making them more responsive and empathetic. For customer service, it could lead to tailored communications and solutions, boosting satisfaction and loyalty. In security, it could help identify distress or suspicious behaviors more effectively. Recommendations for Implementation Integration with Interactive Systems: Deploy the model in platforms for enhanced user interaction, such as customer service kiosks or online help desks. Enhancement of Security Systems: Use the model for improved surveillance and monitoring, flagging unusual or suspicious behavior. Actionables for Stakeholders:

Deployment and Training: Implement the model on targeted platforms and train staff to use it effectively. Continuous Model Evaluation and Improvement: Regularly review and update the model with new data to ensure its accuracy and relevance. Privacy and Ethical Considerations: Set guidelines for ethical use, respecting privacy and data protection laws.

Costs: The training of these models required backend gpu services provided through google. For a larger scale version the costs would go up accordingly. This project alone used 200 compute units over the period of a week training and re-training models. For a more complex model that would be required to scale up the services explored here cost analysis should be taken into consideration depending on the rate of scaling.

Available: 16.86 compute units Usage rate: approximately 2.05 per hour



To conclude, Computer Vision, is where all the code and software actually make contact with our world. Unlike robotics, for computer vision all that is required is a simple camera, giving the technology the ability to reach and touch most of the population on the planet depening on the use case.

Thank you -- Robert A, Sloan



CNN_MODEL_1 PERFORMACE¶


In [ ]:
plot_model_performance_and_confusion_matrix(cnn_model_1, history, test_generator,
                                            metrics_function=lambda true, pred: print(classification_report(true, pred)))
4/4 [==============================] - 0s 15ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.87      0.81      0.84        32
           1       0.65      0.88      0.75        32
           2       0.81      0.69      0.75        32
           3       0.96      0.84      0.90        32

    accuracy                           0.80       128
   macro avg       0.82      0.80      0.81       128
weighted avg       0.82      0.80      0.81       128


CNN_MODEL_2 PERFORMANCE¶


In [ ]:
 
4/4 [==============================] - 0s 10ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.82      0.84      0.83        32
           1       0.72      0.81      0.76        32
           2       0.79      0.69      0.73        32
           3       0.97      0.94      0.95        32

    accuracy                           0.82       128
   macro avg       0.82      0.82      0.82       128
weighted avg       0.82      0.82      0.82       128

4/4 [==============================] - 0s 11ms/step - loss: 0.5187 - accuracy: 0.8203
Test Loss: 0.5187498927116394
Test Accuracy: 0.8203125

Test Loss: CNN_MODEL_updated

Test Loss: 0.5187

Test Accuracy: 82.0312%

In [ ]:
 

VGG_16 MODEL PERFORMANCE¶


In [ ]:
 
4/4 [==============================] - 0s 11ms/step
Confusion Matrix:
[[25  6  0  1]
 [ 0 28  3  1]
 [ 3  8 18  3]
 [ 0  2  2 28]]
4/4 [==============================] - 0s 13ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
              precision    recall  f1-score   support

           0       0.89      0.78      0.83        32
           1       0.64      0.88      0.74        32
           2       0.78      0.56      0.65        32
           3       0.85      0.88      0.86        32

    accuracy                           0.77       128
   macro avg       0.79      0.77      0.77       128
weighted avg       0.79      0.77      0.77       128


RESNET MODEL PERFORMANCE¶


In [ ]:
 
4/4 [==============================] - 1s 20ms/step
Confusion Matrix:
[[26  2  2  2]
 [ 2 25  5  0]
 [ 1 12 17  2]
 [ 0  3  1 28]]
4/4 [==============================] - 0s 22ms/step
dict_keys(['happy', 'neutral', 'sad', 'surprise'])
              precision    recall  f1-score   support

           0       0.90      0.81      0.85        32
           1       0.60      0.78      0.68        32
           2       0.68      0.53      0.60        32
           3       0.88      0.88      0.88        32

    accuracy                           0.75       128
   macro avg       0.76      0.75      0.75       128
weighted avg       0.76      0.75      0.75       128


EFFICIENT NET PERFORMANCE¶


In [244]:
 
4/4 [==============================] - 0s 12ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.67      0.81      0.73        32
           1       0.57      0.72      0.64        32
           2       0.54      0.44      0.48        32
           3       0.87      0.62      0.73        32

    accuracy                           0.65       128
   macro avg       0.66      0.65      0.65       128
weighted avg       0.66      0.65      0.65       128

4/4 [==============================] - 0s 18ms/step - loss: 0.8464 - accuracy: 0.6484
Test Loss: 0.84635990858078
Test Accuracy: 0.6484375

Test Loss: efficient_model2

Test Loss: 0.85

Test Accuracy: 64.84%


THIRD DEEP CONVOLUTIONAL NEURAL NETOWRK WITH 5 CNN BLOCKS: PERFORMANCE¶


In [ ]:
 
4/4 [==============================] - 0s 17ms/step
Class names in the dataset: ['happy', 'neutral', 'sad', 'surprise']
              precision    recall  f1-score   support

           0       0.86      0.78      0.82        32
           1       0.65      0.75      0.70        32
           2       0.70      0.66      0.68        32
           3       0.84      0.84      0.84        32

    accuracy                           0.76       128
   macro avg       0.76      0.76      0.76       128
weighted avg       0.76      0.76      0.76       128

4/4 [==============================] - 0s 20ms/step - loss: 0.6133 - accuracy: 0.7578
Test Loss: 0.6132991909980774
Test Accuracy: 0.7578125

Test Loss: CNN_MODEL_4

Test Loss: 0.6133

Test Accuracy: 75.7812%


FINAL EXPERIMENTAL SIAMESE NETWORK PERFORMANCE¶


In [ ]:
 
4/4 [==============================] - 0s 16ms/step
              precision    recall  f1-score   support

           0       0.83      0.91      0.87        32
           1       0.69      0.75      0.72        32
           2       0.72      0.66      0.69        32
           3       0.97      0.88      0.92        32

    accuracy                           0.80       128
   macro avg       0.80      0.80      0.80       128
weighted avg       0.80      0.80      0.80       128

In [ ]:
!tensorflowjs_converter --input_format=keras /content/cnn_model_4.h5 /content/siamese_model_1
2024-02-22 05:12:35.043733: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-22 05:12:35.043782: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-22 05:12:35.044911: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-22 05:12:36.086643: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
In [241]:
# prompt: using the nbconvert tool convert the notebook to html

!jupyter nbconvert --to html "/content/cnn_model_4.ipynb"
[NbConvertApp] WARNING | pattern '/content/notebook.ipynb' matched no files
This application is used to convert notebook files (*.ipynb)
        to various other formats.

        WARNING: THE COMMANDLINE INTERFACE MAY CHANGE IN FUTURE RELEASES.

Options
=======
The options below are convenience aliases to configurable class-options,
as listed in the "Equivalent to" description-line of the aliases.
To see all configurable class-options for some <cmd>, use:
    <cmd> --help-all

--debug
    set log level to logging.DEBUG (maximize logging output)
    Equivalent to: [--Application.log_level=10]
--show-config
    Show the application's configuration (human-readable format)
    Equivalent to: [--Application.show_config=True]
--show-config-json
    Show the application's configuration (json format)
    Equivalent to: [--Application.show_config_json=True]
--generate-config
    generate default config file
    Equivalent to: [--JupyterApp.generate_config=True]
-y
    Answer yes to any questions instead of prompting.
    Equivalent to: [--JupyterApp.answer_yes=True]
--execute
    Execute the notebook prior to export.
    Equivalent to: [--ExecutePreprocessor.enabled=True]
--allow-errors
    Continue notebook execution even if one of the cells throws an error and include the error message in the cell output (the default behaviour is to abort conversion). This flag is only relevant if '--execute' was specified, too.
    Equivalent to: [--ExecutePreprocessor.allow_errors=True]
--stdin
    read a single notebook file from stdin. Write the resulting notebook with default basename 'notebook.*'
    Equivalent to: [--NbConvertApp.from_stdin=True]
--stdout
    Write notebook output to stdout instead of files.
    Equivalent to: [--NbConvertApp.writer_class=StdoutWriter]
--inplace
    Run nbconvert in place, overwriting the existing notebook (only
            relevant when converting to notebook format)
    Equivalent to: [--NbConvertApp.use_output_suffix=False --NbConvertApp.export_format=notebook --FilesWriter.build_directory=]
--clear-output
    Clear output of current file and save in place,
            overwriting the existing notebook.
    Equivalent to: [--NbConvertApp.use_output_suffix=False --NbConvertApp.export_format=notebook --FilesWriter.build_directory= --ClearOutputPreprocessor.enabled=True]
--no-prompt
    Exclude input and output prompts from converted document.
    Equivalent to: [--TemplateExporter.exclude_input_prompt=True --TemplateExporter.exclude_output_prompt=True]
--no-input
    Exclude input cells and output prompts from converted document.
            This mode is ideal for generating code-free reports.
    Equivalent to: [--TemplateExporter.exclude_output_prompt=True --TemplateExporter.exclude_input=True --TemplateExporter.exclude_input_prompt=True]
--allow-chromium-download
    Whether to allow downloading chromium if no suitable version is found on the system.
    Equivalent to: [--WebPDFExporter.allow_chromium_download=True]
--disable-chromium-sandbox
    Disable chromium security sandbox when converting to PDF..
    Equivalent to: [--WebPDFExporter.disable_sandbox=True]
--show-input
    Shows code input. This flag is only useful for dejavu users.
    Equivalent to: [--TemplateExporter.exclude_input=False]
--embed-images
    Embed the images as base64 dataurls in the output. This flag is only useful for the HTML/WebPDF/Slides exports.
    Equivalent to: [--HTMLExporter.embed_images=True]
--sanitize-html
    Whether the HTML in Markdown cells and cell outputs should be sanitized..
    Equivalent to: [--HTMLExporter.sanitize_html=True]
--log-level=<Enum>
    Set the log level by value or name.
    Choices: any of [0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']
    Default: 30
    Equivalent to: [--Application.log_level]
--config=<Unicode>
    Full path of a config file.
    Default: ''
    Equivalent to: [--JupyterApp.config_file]
--to=<Unicode>
    The export format to be used, either one of the built-in formats
            ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides', 'webpdf']
            or a dotted object name that represents the import path for an
            ``Exporter`` class
    Default: ''
    Equivalent to: [--NbConvertApp.export_format]
--template=<Unicode>
    Name of the template to use
    Default: ''
    Equivalent to: [--TemplateExporter.template_name]
--template-file=<Unicode>
    Name of the template file to use
    Default: None
    Equivalent to: [--TemplateExporter.template_file]
--theme=<Unicode>
    Template specific theme(e.g. the name of a JupyterLab CSS theme distributed
    as prebuilt extension for the lab template)
    Default: 'light'
    Equivalent to: [--HTMLExporter.theme]
--sanitize_html=<Bool>
    Whether the HTML in Markdown cells and cell outputs should be sanitized.This
    should be set to True by nbviewer or similar tools.
    Default: False
    Equivalent to: [--HTMLExporter.sanitize_html]
--writer=<DottedObjectName>
    Writer class used to write the
                                        results of the conversion
    Default: 'FilesWriter'
    Equivalent to: [--NbConvertApp.writer_class]
--post=<DottedOrNone>
    PostProcessor class used to write the
                                        results of the conversion
    Default: ''
    Equivalent to: [--NbConvertApp.postprocessor_class]
--output=<Unicode>
    overwrite base name use for output files.
                can only be used when converting one notebook at a time.
    Default: ''
    Equivalent to: [--NbConvertApp.output_base]
--output-dir=<Unicode>
    Directory to write output(s) to. Defaults
                                  to output to the directory of each notebook. To recover
                                  previous default behaviour (outputting to the current
                                  working directory) use . as the flag value.
    Default: ''
    Equivalent to: [--FilesWriter.build_directory]
--reveal-prefix=<Unicode>
    The URL prefix for reveal.js (version 3.x).
            This defaults to the reveal CDN, but can be any url pointing to a copy
            of reveal.js.
            For speaker notes to work, this must be a relative path to a local
            copy of reveal.js: e.g., "reveal.js".
            If a relative path is given, it must be a subdirectory of the
            current directory (from which the server is run).
            See the usage documentation
            (https://nbconvert.readthedocs.io/en/latest/usage.html#reveal-js-html-slideshow)
            for more details.
    Default: ''
    Equivalent to: [--SlidesExporter.reveal_url_prefix]
--nbformat=<Enum>
    The nbformat version to write.
            Use this to downgrade notebooks.
    Choices: any of [1, 2, 3, 4]
    Default: 4
    Equivalent to: [--NotebookExporter.nbformat_version]

Examples
--------

    The simplest way to use nbconvert is

            > jupyter nbconvert mynotebook.ipynb --to html

            Options include ['asciidoc', 'custom', 'html', 'latex', 'markdown', 'notebook', 'pdf', 'python', 'rst', 'script', 'slides', 'webpdf'].

            > jupyter nbconvert --to latex mynotebook.ipynb

            Both HTML and LaTeX support multiple output templates. LaTeX includes
            'base', 'article' and 'report'.  HTML includes 'basic', 'lab' and
            'classic'. You can specify the flavor of the format used.

            > jupyter nbconvert --to html --template lab mynotebook.ipynb

            You can also pipe the output to stdout, rather than a file

            > jupyter nbconvert mynotebook.ipynb --stdout

            PDF is generated via latex

            > jupyter nbconvert mynotebook.ipynb --to pdf

            You can get (and serve) a Reveal.js-powered slideshow

            > jupyter nbconvert myslides.ipynb --to slides --post serve

            Multiple notebooks can be given at the command line in a couple of
            different ways:

            > jupyter nbconvert notebook*.ipynb
            > jupyter nbconvert notebook1.ipynb notebook2.ipynb

            or you can specify the notebooks list in a config file, containing::

                c.NbConvertApp.notebooks = ["my_notebook.ipynb"]

            > jupyter nbconvert --config mycfg.py

To see all available configurables, use `--help-all`.